mlsanityData to 7 Oct 2026

Document understanding and multimodal RAG

480 papers0 this weekMedium confidence in this grouping: mostly one topic, with blurry edgesMedium confidence in this grouping: mostly one topic, with blurry edges · About

Keywords: retrieval, document, multimodal, visual, ocr

Key facts. Document understanding and multimodal RAG: 480 papers on arXiv since 2021 (data through 7 Oct 2026). 27 of its 480 mapped papers (6%) cite “TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document” (2024, arXiv 2403.04473). 20 of its 480 mapped papers (4%) cite “OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models” (2023, arXiv 2305.07895). 16 new papers 17 Sep–7 Oct.

What’s new

17 Sep – 7 Oct 2026

16 new papers, 17 Sep – 7 Oct; 0 this week.

  1. Earlier · 16
  2. OpticalRec: Unified Optical Vision-Language Representation for Multimodal Recommendation also fits: VLM benchmarks and visual reasoning
  3. CLIMB: Confidence-Guided Complementary Evidence for Multimodal Retrieval-Augmented Generation
  4. Found but Not Read: When Extracted Text Closes the Retrieval-Reading Gap in Document Vision-Language Models
  5. AMBER: Multi-View Adaptive Budget Allocation for Listwise Vision-Language Reranking
  6. Walking the Embedding Space: Datastore Extraction from Multimodal RAG also fits: MLLM jailbreaks and adversarial robustness
  7. Spherical Interpolation for Backward-Compatible Multimodal Representations
  8. ResComEmb: Effective and Efficient Multimodal Embedding via Residual Homogeneity Compression also fits: MLLM training and visual instruction tuning
  9. Xiaomi-OCR-0 Technical Report
  10. RenderRank: Learning to Rerank Text with Compressed Visual Tokens
  11. ColNanoVDR: Document-Free Query Distillation for Multi-Vector Visual Document Retrieval via Optimal Transport
All 16
  1. RidgeRank: Efficient Visual Document Reranking via Score Fusion and a Shallow Linear Readout
  2. Learning Multimodal Embeddings with Evidence-Aligned Readout also fits: MLLM training and visual instruction tuning
  3. Concepts Complement Dense Semantics: Learning Compact Sparse Spaces for Text-Image Retrieval
  4. Sorry Robot, Happy Human: Vision-Language Models Read Only One of Two Legible Typographic Layers also fits: VLM benchmarks and visual reasoning
  5. An Empirical Study of VLM Pipelines for Long-Document QA
  6. A Free Lunch? Adapting PP-OCRv6 for Historical Text Recognition

Foundations

Most cited by papers here

  1. TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document2024cited by 27 here
  2. OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models2023cited by 20 here
  3. ColPali: Efficient Document Retrieval with Vision Language Models2024cited by 17 here
  4. OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning2024cited by 17 here
  5. mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding2023cited by 16 here

Lineage

The papers this direction builds on, oldest first.

Each paper on the line cites the one before.

  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 36 here
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 38 here
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 21 here
  4. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 13 here
  5. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 30 here
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 51 here
  7. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 17 here
  8. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 17 here
16 more it builds on, not in this chain
  1. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 20 here
  2. 2023A Survey on Multimodal Large Language Modelscited by 12 here
  3. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 16 here
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 18 here
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 45 here
  6. 2023UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Modelcited by 14 here
  7. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 20 here
  8. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 37 here
  9. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 27 here
  10. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 15 here
  11. 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 17 here
  12. 2024E5-V: Universal Embeddings with Multimodal Large Language Modelscited by 13 here
  13. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 27 here
  14. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 13 here
  15. 2024MM-Embed: Universal Multimodal Retrieval with Multimodal LLMscited by 13 here
  16. 2025Qwen3-VL Technical Reportcited by 15 here

The 24 most cited of 593.

Browse

30 of 480, ranked 6–35

  1. 6E5-V: Universal Embeddings with Multimodal Large Language Models202413
  2. 7VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks202413
  3. 8MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs202413
  4. 9olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models202512
  5. 10Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs202411
  6. 11LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant202411
  7. 12CC-OCR: A Comprehensive and Challenging OCR Benchmark for Evaluating Large Multimodal Models in Literacy202411
  8. 13LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding20249
  9. 14EchoSight: Advancing Visual-Language Models with Wiki Knowledge20249
  10. 15DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understanding20239
All 30
  1. 16GME: Improving Universal Multimodal Retrieval by Multimodal LLMs20248
  2. 17REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory20228
  3. 18MMSearch-R1: Incentivizing LMMs to Search20257
  4. 19LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locating20247
  5. 20Kaleido-BERT: Vision-Language Pre-training on Fashion Domain20217
  6. 21SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversion20256
  7. 22Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs20256
  8. 23PaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Model20256
  9. 24DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimming20246
  10. 25Ocean-OCR: Towards General OCR Application via a Vision-Language Model20255
  11. 26Docopilot: Improving Multimodal Models for Document-Level Understanding20255
  12. 27HunyuanOCR Technical Report20255
  13. 28Harnessing Multimodal Large Language Models for Multimodal Sequential Recommendation20245
  14. 29RoRA-VLM: Robust Retrieval-Augmented Vision Language Models20245
  15. 30DocVLM: Make Your VLM an Efficient Reader20245
  16. 31Retrieval-Augmented Multimodal Language Modeling20225
  17. 32A Token-level Text Image Foundation Model for Document Understanding20254
  18. 33POINTS-Reader: Distillation-Free Adaptation of Vision-Language Models for Document Conversion20254
  19. 34MinerU2.5: A Decoupled Vision-Language Model for Efficient High-Resolution Document Parsing20254
  20. 35DeepMMSearch-R1: Empowering Multimodal LLMs in Multimodal Web Search20254