mlsanityData to 7 Oct 2026

UniDoc: A Universal Large Multimodal Model for Simultaneous Text Detection, Recognition, Spotting and Understanding

arXiv 2308.11592 ↗First posted 19 Aug 2023cs.AI cs.CLcited by 74

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

In the era of Large Language Models (LLMs), tremendous strides have been made in the field of multimodal understanding. However, existing advanced algorithms are limited to effectively utilizing the immense representation capabilities and rich world knowledge inherent to these large pre-trained models, and the beneficial connections among tasks within the context of text-rich scenarios have not been sufficiently explored. In this work, we introduce UniDoc, a novel multimodal model equipped with text detection and recognition capabilities, which are deficient in existing approaches. Moreover, UniDoc capitalizes on the beneficial interactions among tasks to enhance the performance of each individual task. To implement UniDoc, we perform unified multimodal instruct tuning on the contributed large-scale instruction following datasets. Quantitative and qualitative experimental results show that UniDoc sets state-of-the-art scores across multiple challenging benchmarks. To the best of our knowledge, this is the first large multimodal model capable of simultaneous text detection, recognition, spotting, and understanding.

Key facts. Cited by 74 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 21 of its 30 references that are in the map. First posted on arXiv 19 Aug 2023.

Builds on

21 of 30 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun

Other references · 20

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  11. 2019Towards VQA Models That Can Readcited by 1,696
  12. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
8 more
  1. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  2. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  3. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 167
  4. 2017Total-Text: A Comprehensive Dataset for Scene Text Detection and Recognitioncited by 161
  5. 2021From Two to One: A New Scene Text Recognizer with Visual Language Modeling Networkcited by 97
  6. 2017Detecting Oriented Text in Natural Images by Linking Segmentscited by 80
  7. 2017Towards End-to-end Text Spotting with Convolutional Recurrent Neural Networkscited by 56
  8. 2022Toward Understanding WordArt: Corner-Guided Transformer for Scene Text Recognitioncited by 45

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 23; 2025: 19; 2026: 28.

74 papers.

Sort:

By area:

  1. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 173
  2. 2023UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Modelcited by 143
  3. 2023DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understandingcited by 106
  4. 2024MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answeringcited by 93
  5. 2025Dolphin: Document Image Parsing via Heterogeneous Anchor Promptingcited by 78
  6. 2024OmniParser: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognitioncited by 74
  7. 2024TabPedia: Towards Comprehensive Visual Table Understanding with Concept Synergycited by 70
  8. 2024TextSquare: Scaling up Text-Centric Visual Instruction Tuningcited by 62
  9. 2024A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understandingcited by 57
  10. 2024Harmonizing Visual Text Comprehension and Generationcited by 54
  11. 2024TextCoT: Zoom In for Enhanced Multimodal Text-Rich Image Understandingcited by 37
  12. 2025Representation Learning for Tabular Data: A Comprehensive Surveycited by 36
38 more
  1. 2024DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimmingcited by 35
  2. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  3. 2025Prolonged Reasoning Is Not All You Need: Certainty-Based Adaptive Routing for Efficient LLM/MLLM Reasoningcited by 34
  4. 2023mPLUG-PaperOwl: Scientific Diagram Analysis with the Multimodal Large Language Modelcited by 31
  5. 2024Enhancing Visual Document Understanding with Contrastive Learning in Large Visual-Language Modelscited by 30
  6. 2024TextHawk: Exploring Efficient Fine-Grained Perception of Multimodal Large Language Modelscited by 27
  7. 2024Mini-Monkey: Alleviating the Semantic Sawtooth Effect for Lightweight MLLMs via Complementary Image Pyramidcited by 26
  8. 2024MCTBench: Multimodal Cognition towards Text-Rich Visual Scenes Benchmarkcited by 26
  9. 2024DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understandingcited by 23
  10. 2025OmniParser V2: Structured-Points-of-Thought for Unified Visual Text Parsing and Its Generality to Multimodal Large Language Modelscited by 22
  11. 2026Dolphin-v2: Universal Document Parsing via Scalable Anchor Promptingcited by 20
  12. 2025SETransformer: A Hybrid Attention-Based Architecture for Robust Human Activity Recognitioncited by 19
  13. 2025Towards Visual Text Grounding of Multimodal Large Language Modelcited by 18
  14. 2024Multi-Pair Temporal Sentence Grounding via Multi-Thread Knowledge Transfer Networkcited by 15
  15. 2024PDF-WuKong: A Large Multimodal Model for Efficient Long PDF Reading with End-to-End Sparse Samplingcited by 14
  16. 2025Multimodal Tabular Reasoning with Privileged Structured Informationcited by 14
  17. 2025A Token-level Text Image Foundation Model for Document Understandingcited by 13
  18. 2023UPOCR: Towards Unified Pixel-Level OCR Interfacecited by 12
  19. 2024LAPDoc: Layout-Aware Prompting for Documentscited by 9
  20. 2024Lumos : Empowering Multimodal LLMs with Scene Text Recognitioncited by 8
  21. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Reviewcited by 8
  22. 2025A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trendscited by 8
  23. 2024PaDeLLM-NER: Parallel Decoding in Large Language Models for Named Entity Recognitioncited by 6
  24. 2025Survey on Question Answering over Visually Rich Documents: Methods, Challenges, and Trendscited by 4
  25. 2025DocSAM: Unified Document Image Segmentation via Query Decomposition and Heterogeneous Mixed Learningcited by 4
  26. 2026Beyond Human Annotation: Recent Advances in Data Generation Methods for Document Intelligencecited by 4
  27. 2024MindBench: A Comprehensive Benchmark for Mind Map Structure Recognition and Analysiscited by 3
  28. 2024Platypus: A Generalized Specialist Model for Reading Text in Various Formscited by 2
  29. 2025DREAM: Document Reconstruction via End-to-end Autoregressive Modelcited by 2
  30. 2025Mind the (Language) Gap: Towards Probing Numerical and Cross-Lingual Limits of LVLMscited by 2
  31. 2024AdaptVision: Dynamic Input Scaling in MLLMs for Versatile Scene Understandingcited by 1
  32. 2025TDRI: Two-Phase Dialogue Refinement and Co-Adaptation for Interactive Image Generationcited by 1
  33. 2026SEA-Vision: A Multilingual Benchmark for Comprehensive Document and Scene Text Understanding in Southeast Asiacited by 1
  34. 2024Hear the Scene: Audio-Enhanced Text Spottingcited by 0
  35. 2025First-place Solution for Streetscape Shop Sign Recognition Competitioncited by 0
  36. 2025Cross-Modal Synergies: Unveiling the Potential of Motion-Aware Fusion Networks in Handling Dynamic and Static ReID Scenarioscited by 0
  37. 2025OMR-Diffusion:Optimizing Multi-Round Enhanced Training in Diffusion Models for Improved Intent Understandingcited by 0
  38. 2026LPCAN: Lightweight Pyramid Cross-Attention Network for Rail Surface Defect Detection Using RGB-D Datacited by 0

50 most cited of 74 All 74 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs
  2. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey
  3. 2024LLaVA-Read: Enhancing Reading Ability of Multimodal Language Models
  4. 2025UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG
  5. 2025LMM-Det: Make Large Multimodal Models Excel in Object Detection
  6. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning
  7. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  8. 2025UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generation