mlsanityData to 14 Oct 2026

KOSMOS-2.5: A Multimodal Literate Model

arXiv 2309.11419 ↗First posted 20 Sep 2023cs.CL cs.CVcited by 96

Not on the map. Why?

Cited by 47 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The automatic reading of text-intensive images represents a significant advancement toward achieving Artificial General Intelligence (AGI). In this paper we present KOSMOS-2.5, a multimodal literate model for machine reading of text-intensive images. Pre-trained on a large-scale corpus of text-intensive images, KOSMOS-2.5 excels in two distinct yet complementary transcription tasks: (1) generating spatially-aware text blocks, where each block of text is assigned spatial coordinates within the image, and (2) producing structured text output that captures both style and structure in markdown format. This unified multimodal literate capability is achieved through a shared decoder-only autoregressive Transformer architecture and task-specific prompts. Building on this foundation, we fine-tune KOSMOS-2.5 for document understanding tasks, resulting in a document understanding generalist named KOSMOS-2.5-CHAT. Additionally, a large corpus of 357.4 million document pages spanning diverse domains was curated for pre-training. We evaluate KOSMOS-2.5 on two newly proposed benchmarks, OCREval and MarkdownEval, for document-level text recognition and image-to-markdown generation, demonstrating impressive literate capabilities comparable to GPT-4o. KOSMOS-2.5-CHAT achieves performance comparable to other state-of-the-art generalists that are five times larger (1.3B vs. 7B) across nine text-rich visual question answering benchmarks. Models and code have been available at urlhttps://aka.ms/kosmos25.

Key facts. Cited by 96 papers in the map (data through 14 Oct 2026). Builds on 75 of its 127 references that are in the map. First posted on arXiv 20 Sep 2023.

Builds on

75 of 127 references.

References · 75

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  12. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
38 more
  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  3. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  4. 2019Towards VQA Models That Can Readcited by 1,718
  5. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  6. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  7. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  8. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  9. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  10. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  11. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  12. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  13. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  14. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  15. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  16. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 553
  17. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  18. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  19. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  20. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 515
  21. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  22. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 485
  23. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 428
  24. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 408
  25. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  26. 2016Synthetic Data for Text Localisation in Natural Imagescited by 396
  27. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 366
  28. 2019FUNSD: A Dataset for Form Understanding in Noisy Scanned Documentscited by 349
  29. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  30. 2021TrOCR: Transformer-based Optical Character Recognition with Pre-trained Modelscited by 279
  31. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 274
  32. 2021ICDAR2019 Competition on Scanned Receipt OCR and Information Extractioncited by 256
  33. 2021DocFormer: End-to-End Transformer for Document Understandingcited by 256
  34. 2023Nougat: Neural Optical Understanding for Academic Documentscited by 245
  35. 2023TextDiffuser: Diffusion Models as Text Painterscited by 229
  36. 2014Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognitioncited by 214
  37. 2023Any-to-Any Generation via Composable Diffusioncited by 204
  38. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 39; 2025: 32; 2026: 18.

96 papers.

Sort:

By area:

  1. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  2. 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
  3. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 311
  4. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  5. 2024MinerU: An Open-Source Solution for Precise Document Content Extractioncited by 267
  6. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  7. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 159
  8. 2023GSVA: Generalized Segmentation via Multimodal Large Language Modelscited by 154
  9. 2024Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineeringcited by 152
  10. 2023TextDiffuser-2: Unleashing the Power of Language Models for Text Renderingcited by 140
  11. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 137
  12. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 134
38 more
  1. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 131
  2. 2025EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videoscited by 128
  3. 2023DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understandingcited by 106
  4. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
  5. 2024CC-OCR: A Comprehensive and Challenging OCR Benchmark for Evaluating Large Multimodal Models in Literacycited by 81
  6. 2024MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizercited by 80
  7. 2024ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuningcited by 79
  8. 2025Dolphin: Document Image Parsing via Heterogeneous Anchor Promptingcited by 79
  9. 2024An Empirical Analysis on Spatial Reasoning Capabilities of Large Multimodal Modelscited by 50
  10. 2024AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information?cited by 47
  11. 2024SAR3D: Autoregressive 3D Object Generation and Understanding via Multi-scale 3D VQVAEcited by 39
  12. 2025Open3D-VQA: A Benchmark for Comprehensive Spatial Reasoning with Multimodal Large Language Model in Open Spacecited by 39
  13. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  14. 2024DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Language Modelscited by 36
  15. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  16. 20253D Aware Region Prompted Vision Language Modelcited by 33
  17. 2024TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokenscited by 32
  18. 2024ChatRex: Taming Multimodal LLM for Joint Perception and Understandingcited by 30
  19. 2024V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encodingcited by 28
  20. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  21. 2023Towards Improving Document Understanding: An Exploration on Text-Grounding via MLLMscited by 25
  22. 2025OmniParser V2: Structured-Points-of-Thought for Unified Visual Text Parsing and Its Generality to Multimodal Large Language Modelscited by 23
  23. 2026Dolphin-v2: Universal Document Parsing via Scalable Anchor Promptingcited by 22
  24. 2024The Evolution of Multimodal Model Architecturescited by 21
  25. 2023T-Rex: Counting by Visual Promptingcited by 20
  26. 2025Marten: Visual Question Answering with Mask Generation for Multi-modal Document Understandingcited by 20
  27. 2025Towards Visual Text Grounding of Multimodal Large Language Modelcited by 18
  28. 2024READoc: A Unified Benchmark for Realistic Document Structured Extractioncited by 15
  29. 2025A Token-level Text Image Foundation Model for Document Understandingcited by 14
  30. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  31. 2025UGround: Towards Unified Visual Grounding with Unrolled Transformerscited by 14
  32. 2024VisualCritic: Making LMMs Perceive Visual Quality Like Humanscited by 13
  33. 2023UPOCR: Towards Unified Pixel-Level OCR Interfacecited by 12
  34. 2025Seeing is Believing? Mitigating OCR Hallucinations in Multimodal Large Language Modelscited by 12
  35. 2024StrucTexTv3: An Efficient Vision-Language Model for Text-rich Image Perception, Comprehension, and Beyondcited by 11
  36. 2024DOGR: Towards Versatile Visual Document Grounding and Referringcited by 10
  37. 2024LAPDoc: Layout-Aware Prompting for Documentscited by 9
  38. 2025'Eclair -- Extracting Content and Layout with Integrated Reading Order for Documentscited by 9

50 most cited of 96 All 96 (JSON)