mlsanityData to 14 Oct 2026

UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model

arXiv 2310.05126 ↗First posted 8 Oct 2023cs.CV cs.AIcited by 143

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Text is ubiquitous in our visual world, conveying crucial information, such as in documents, websites, and everyday photographs. In this work, we propose UReader, a first exploration of universal OCR-free visually-situated language understanding based on the Multimodal Large Language Model (MLLM). By leveraging the shallow text recognition ability of the MLLM, we only finetuned 1.2% parameters and the training cost is much lower than previous work following domain-specific pretraining and finetuning paradigms. Concretely, UReader is jointly finetuned on a wide range of Visually-situated Language Understanding tasks via a unified instruction format. To enhance the visual text and semantic understanding, we further apply two auxiliary tasks with the same format, namely text reading and key points generation tasks. We design a shape-adaptive cropping module before the encoder-decoder architecture of MLLM to leverage the frozen low-resolution vision encoder for processing high-resolution images. Without downstream finetuning, our single model achieves state-of-the-art ocr-free performance in 8 out of 10 visually-situated language understanding tasks, across 5 domains: documents, tables, charts, natural images, and webpage screenshots. Codes and instruction-tuning datasets will be released.

Key facts. Cited by 143 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 47 references that are in the map. First posted on arXiv 8 Oct 2023.

Builds on

34 of 47 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023UniDoc: A Universal Large Multimodal Model for Simultaneous Text Detection, Recognition, Spotting and Understanding19 Aug
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  8. 2019Towards VQA Models That Can Readcited by 1,718
  9. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  10. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  11. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  12. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
20 more
  1. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 515
  2. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 485
  3. 2021InfographicVQAcited by 437
  4. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 428
  5. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 408
  6. 2021OCR-free Document Understanding Transformercited by 407
  7. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  8. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 366
  9. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 340
  10. 2019Scene Text Visual Question Answeringcited by 331
  11. 2019PlotQA: Reasoning over Scientific Plotscited by 321
  12. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 319
  13. 2021VisualMRC: Machine Reading Comprehension on Document Imagescited by 151
  14. 2022Unifying Vision, Text, and Layout for Universal Document Processingcited by 140
  15. 2021WebSRC: A Dataset for Web-Based Structural Reading Comprehensioncited by 100
  16. 2020TAP: Text-Aware Pre-training for Text-VQA and Text-Captioncited by 95
  17. 2021Kleister: Key Information Extraction Datasets Involving Long Documents with Complex Layoutscited by 82
  18. 2022End-to-end Document Recognition and Understanding with Dessurtcited by 73
  19. 2023MPMQA: Multimodal Question Answering on Product Manualscited by 10
  20. 2021Question-controlled Text-aware Image Captioningcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 57; 2025: 55; 2026: 25.

143 papers.

Sort:

By area:

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  2. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  3. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  4. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  5. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 197
  6. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 195
  7. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  8. 2024General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Modelcited by 180
  9. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 174
  10. 2024Building and better understanding vision-language models: insights and future directionscited by 173
  11. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 173
  12. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 172
38 more
  1. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 159
  2. 2025RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concretecited by 135
  3. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 125
  4. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Modelcited by 121
  5. 2023ChartBench: A Benchmark for Complex Visual Reasoning in Chartscited by 120
  6. 2023DocLLM: A layout-aware generative language model for multimodal document understandingcited by 116
  7. 2024mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understandingcited by 114
  8. 2023DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understandingcited by 106
  9. 2024Unveiling Encoder-Free Vision-Language Modelscited by 94
  10. 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMscited by 84
  11. 2024DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perceptioncited by 73
  12. 2024TabPedia: Towards Comprehensive Visual Table Understanding with Concept Synergycited by 71
  13. 2025Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Modelscited by 70
  14. 2024TextSquare: Scaling up Text-Centric Visual Instruction Tuningcited by 62
  15. 2024Focus Anywhere for Fine-grained Multi-page Document Understandingcited by 62
  16. 2025VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documentscited by 59
  17. 2025GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensingcited by 57
  18. 2024Harmonizing Visual Text Comprehension and Generationcited by 54
  19. 2025SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversioncited by 53
  20. 2025FineVision: Open Data Is All You Needcited by 53
  21. 2025OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generationcited by 50
  22. 2025UniToken: Harmonizing Multimodal Understanding and Generation through Unified Visual Encodingcited by 49
  23. 2024MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Modelscited by 46
  24. 2024EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotionscited by 45
  25. 2024TextCoT: Zoom In for Enhanced Multimodal Text-Rich Image Understandingcited by 38
  26. 2024HRVDA: High-Resolution Visual Document Assistantcited by 37
  27. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  28. 2024DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimmingcited by 35
  29. 2024TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokenscited by 32
  30. 2025We-Math 2.0: A Versatile MathBook System for Incentivizing Visual Mathematical Reasoningcited by 32
  31. 2024Enhancing Visual Document Understanding with Contrastive Learning in Large Visual-Language Modelscited by 30
  32. 2025SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understandingcited by 30
  33. 2024DocVLM: Make Your VLM an Efficient Readercited by 28
  34. 2026LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decodingcited by 28
  35. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  36. 2024ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal Modelscited by 26
  37. 2023Towards Improving Document Understanding: An Exploration on Text-Grounding via MLLMscited by 25
  38. 2024BlueLM-V-3B: Algorithm and System Co-Design for Multimodal Large Language Models on Mobile Devicescited by 25

50 most cited of 143 All 143 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024On Efficient Language and Vision Assistants for Visually-Situated Natural Language Understanding: What Matters in Reading and Reasoning
  2. 2023Visually-Situated Natural Language Understanding with Contrastive Reading Model and Frozen Large Language Models
  3. 2025UniCode2: Cascaded Large-scale Codebooks for Unified Multimodal Understanding and Generation
  4. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review
  5. 2024From Text to Pixel: Advancing Long-Context Understanding in MLLMs
  6. 2025Unified Multimodal Understanding via Byte-Pair Visual Encoding
  7. 2025ReadBench: Measuring the Dense Text Visual Reading Ability of Vision-Language Models
  8. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond