【摘要】 目的 文档智能旨在自动和智能地处理纸质文本信息，包括但不限于表格、表单和发票等，极大便利了信息的电子化管理。然而，传统深度学习方法往往专注于单一任务的优化，限制了它们在处理复杂多变的文档场景时的效能。此外，这些方法需要额外的光学字符识别（optical character recognition,OCR）工具来提取文档中的文字信息，这不仅增加了处理步骤，也可能引入额外的错误。多模态大模型的出现为免去OCR工具统一处理文档信息带来了希望，但是在处理高分辨率的文档图像和应对逐渐增加的视觉标记时，仍然面临着不小的挑战。本文提出一种基于动态分辨率的文档多模态大模型TextLLM，能免OCR工具处理高分辨率的文档图像。方法 基于最新的多模态大模型训练了一个能够处理动态分辨率的文档多模态大模型。在动态分辨率的基础上，提出一种动态特征压缩算法，设置动态的可学习压缩率来获得需要保留的特征长度，通过计算特征相似度来得到重要性特征，以此来聚合关键特征。更进一步，利用大语言模型的注意力机制捕捉与提示词相关的视觉特征部分，根据提示词的注意力分布图筛选出最相关的特征，并保留其周围相关特征。结果 实验在多个数据集上与6种先进方法进行了比较，TextLLM在多个文档理解基准测试中取得了显著的性能提升。在DocVQA、WTQ、ChartQA和TextVQA等数据集上的表现均优于对比模型，分别获得了82.4、37.6、70.8和65.3的分数。此外，在综合评测数据集OCRBench中，模型得分高达601，证明了其在多样化文本相关任务中的适应能力和整体效果。同时也在多个数据集中进行了消融实验以验证算法的有效性，消融实验验证了提出的动态算法能够改善模型效果。结论 本文提出基于动态分辨率的文档大模型TextLLM，并提出动态压缩特征和动态选择的算法来应对多场景的文档。实验结果表明，本文模型优于几种先进的文档大模型，兼具了高效性和准确性。更多还原

【Abstract】 Objective The advancement of document intelligence aims to realize the intelligent processing and interpretation of various document information, such as the processing of structured documents containing tables, forms, and invoices, among others, and text in natural scenes. Traditional deep learning methods excel at certain tasks, but because they are often optimized for a single task, they struggle to adapt to increasingly complex needs and diverse scenarios. When processing document text, optical character recognition(OCR) is usually required to extract text information. However,the processing speed and accuracy decrease. Moreover, the various steps involved in handling the reading of the pipeline text can lead to the accumulation of errors. In addition, relying on ready-made OCR models/APIs introduces additional engineering complexity, limits connections between text and its surrounding context, and can increase computational costs. OCR-free solutions have attracted increasing attention recently to mitigate the disadvantages of external systems before they are understood. With the rise of multimodal large models, the field of document intelligence is undergoing a revolution. These models integrate textual and visual information to enable a more comprehensive and accurate understanding of document content, potentially eliminating reliance on OCR tools. However, these multimodal large models still face challenges, especially when it comes to processing high-resolution document images and dealing with the ever-growing number of visual markers. Previous similar working resolutions are limited by the input size of the encoder, and seeing specific small text is generally difficult. Reducing the image resolution will also greatly increase the number of tokens of the image, which is challenging for memory and space consumption. A large multimodal model based on dynamic resolution is proposed to overcome these challenges. This model is designed to handle high-resolution document images, eliminating the need for OCR tools while being flexible enough to accommodate ever-increasing visual tokens.Method In this study, we improve the latest multimodal large model and train a large document model on the basis of dynamic resolution, called Text LLM. This process mainly involves dynamic adjustment of the image, block processing, visual coding, feature compression, and screening to extract and retain useful information to the maximum extent. First, the area closest to the predefined scaling size is determined on the basis of the original image size for subsequent processing, including scaling processing, image slicing, and global information acquisition. Next, the image blocks are fed into the visual encoder for encoding processing, including window attention mechanisms to recognize and fuse information, and extract local regional visual features and global visual features. Then, the image resampling module is used to process the segmented features, and the compressed visual features are obtained on the basis of the attention mechanism. The compression rate is set dynamically, and discrete compression rates and a learnable parameter are defined to learn the compression rate dynamically. Gumbel softmax is used for end-to-end training to learn the compression rate. According to the compression ratio, the number of features after compression is obtained. The most important features are selected and sorted by calculating the similarity and importance, and these important features are used for further aggregation. Finally, the attention mechanism of the large language model is used to capture the visual features related to the prompt words, the most relevant features are selected according to the attention distribution map of the prompt words, and the surrounding relevant features are retained. The integrated application of these steps helps in effectively processing and utilizing image information and improving the efficiency and accuracy of information extraction.Result Experiments across multiple datasets compared Text LLM with the latest six methods, demonstrating considerably performance improvements in various document understanding benchmarks. It outperforms existing models on datasets such as Doc VQA, WTQ, Chart QA, and Text VQA, achieving scores of 82. 4, 37. 6, 70. 8, and 65. 3, respectively. Furthermore, on the comprehensive OCRBench evaluation dataset, the model achieves a score of 601, proving its adaptability and overall strength in a variety of text-related tasks. Comparative experiments are performed across multiple datasets to verify the effectiveness of the algorithm, and results confirm that the proposed dynamic processing algorithm improved the model’s performance. Several mainstream scenario images are selected for visualization in this paper to further demonstrate the model′s capabilities in text-related tasks across various scenarios. The model is able to accurately recognize text in scene images and documents and answer questions on the basis of its understanding, showing its strong text processing capabilities and adaptability. Introducing dynamic feature compression ensures that the model can autonomously learn the compression rate and sample different compression rates during the learning process, which to some extent serves as data augmentation. Furthermore, incorporating visual feature selection enables the model to focus more on textrelated features. Reducing features further allowed it to achieve improvements across all datasets.Conclusion In this study, we introduce an innovative document large model called Text LLM, which, on the basis of dynamic resolution, combines dynamic compression feature and dynamic selection algorithm. Extensive experimental validation shows that our model not only outperforms several state-of-the-art document large models but also achieves significant improvements in efficiency and accuracy.更多还原