mlsanityData to 7 Oct 2026

SV-RAG: LoRA-Contextualizing Adaptation of MLLMs for Long Document Understanding

arXiv 2411.01106 ↗First posted 2 Nov 2024cs.CVcited by 25

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Multimodal large language models (MLLMs) have recently shown great progress in text-rich image understanding, yet they still struggle with complex, multi-page visually-rich documents. Traditional methods using document parsers for retrieval-augmented generation suffer from performance and efficiency limitations, while directly presenting all pages to MLLMs leads to inefficiencies, especially with lengthy ones. In this work, we present a novel framework named **S**elf-**V**isual **R**etrieval-**A**ugmented **G**eneration (SV-RAG), which can broaden horizons of any MLLM to support long-document understanding. We demonstrate that **MLLMs themselves can be an effective multimodal retriever** to fetch relevant pages and then answer user questions based on these pages. SV-RAG is implemented with two specific MLLM adapters, one for evidence page retrieval and the other for question answering. Empirical results show state-of-the-art performance on public benchmarks, demonstrating the effectiveness of SV-RAG.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 59 of its 69 references that are in the map. First posted on arXiv 2 Nov 2024.

Builds on

59 of 69 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document7 Mar

Other references · 58

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  10. 2020Longformer: The Long-Document Transformercited by 3,362
  11. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  12. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
38 more
  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  2. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  3. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  4. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  5. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  6. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  7. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  8. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  9. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  10. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  11. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  12. 2024PaliGemma: A versatile 3B VLM for transfercited by 679
  13. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 481
  14. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  15. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 466
  16. 2021InfographicVQAcited by 430
  17. 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 402
  18. 2021OCR-free Document Understanding Transformercited by 400
  19. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  20. 2019Scene Text Visual Question Answeringcited by 327
  21. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  22. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  23. 2023FinanceBench: A New Benchmark for Financial Question Answeringcited by 261
  24. 2019Image-based table recognition: data, model, and evaluationcited by 233
  25. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 216
  26. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 204
  27. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 193
  28. 2023SlideVQA: A Dataset for Document Visual Question Answering on Multiple Imagescited by 175
  29. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 174
  30. 2021VisualMRC: Machine Reading Comprehension on Document Imagescited by 150
  31. 2022Unifying Vision, Text, and Layout for Universal Document Processingcited by 140
  32. 2024Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Modelscited by 138
  33. 2023Document Understanding Dataset and Evaluation (DUDE)cited by 132
  34. 2024Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Modelscited by 112
  35. 2024Unifying Multimodal Retrieval via Document Screenshot Embeddingcited by 107
  36. 2022Hierarchical multimodal transformers for Multi-Page DocVQAcited by 102
  37. 2024VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?cited by 98
  38. 2021Document Collection Visual Question Answeringcited by 45

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 9; 2026: 15.

25 papers.

Sort:

By area:

  1. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  2. 2024M-Longdoc: A Benchmark For Multimodal Super-Long Document Understanding And A Retrieval-Aware Tuning Frameworkcited by 36
  3. 2025Resolving Evidence Sparsity: Agentic Context Engineering for Long-Document Understandingcited by 18
  4. 2025LLM/Agent-as-Data-Analyst: A Surveycited by 15
  5. 2025VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understandingcited by 13
  6. 2025Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understandingcited by 9
  7. 2026DocSeeker: Structured Visual Reasoning with Evidence Grounding for Long Document Understandingcited by 6
  8. 2026Doc-V*:Coarse-to-Fine Interactive Visual Reasoning for Multi-Page Document VQAcited by 4
  9. 2026DocPrune:Efficient Document Question Answering via Background, Question, and Comprehension-aware Token Pruningcited by 4
  10. 2025CogDoc: Towards Unified thinking in Documentscited by 3
  11. 2026MoDora: Tree-Based Semi-Structured Document Analysis Systemcited by 3
  12. 2025ALDEN: Reinforcement Learning for Active Navigation and Evidence Gathering in Long Documentscited by 2
13 more
  1. 2025DocSLM: A Small Vision-Language Model for Long Multimodal Document Understandingcited by 2
  2. 2026DocArena: Turning Raw Documents into Controllable Training Environments for Document Search Agentscited by 2
  3. 2026MinerU-Popo: Universal Post-Processing Model for Structured Document Parsingcited by 1
  4. 2026MARDoc: A Memory-Aware Refinement Agent Framework for Multimodal Long Document QAcited by 1
  5. 2026AWM: Answerable Working Memory for Long-Document VQA Agentscited by 1
  6. 2025When Seeing Is not Enough: Revealing the Limits of Active Reasoning in MLLMscited by 0
  7. 2026LITTA: Late-Interaction and Test-Time Alignment for Visually-Grounded Multimodal Retrievalcited by 0
  8. 2026Hydra: Unifying Document Retrieval and Generation in a Single Vision-Language Modelcited by 0
  9. 2026VLD-RAG: Agentic Vision-Language Retrieval-Augmented Generation for Long, Visually-Rich Multi-Page Documentscited by 0
  10. 2026HierDoc: Hierarchical Page-to-Region Evidence Routing for Long-Document Visual Question Answeringcited by 0
  11. 2026DocTrace: Towards Traceable Long Document VQA via Hierarchical Evidence Graph Reasoningcited by 0
  12. 2026InSight-doc: Agentic Visual Perception for Long-Document Understandingcited by 0
  13. 2026EidosDoc: Implicit Structure Encoding for Cost-Effective Semi-Structured Document QAcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Roles of MLLMs in Visually Rich Document Retrieval for RAG: A Survey
  2. 2026Multimodal Graph RAG for Long-range Visually Rich Document Understanding
  3. 2026Hierarchical Evidence-Driven Reasoning for Long Document Understanding
  4. 2025Are We on the Right Way for Assessing Document Retrieval-Augmented Generation?
  5. 2025UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG
  6. 2024Enhanced Multimodal RAG-LLM for Accurate Visual Question Answering
  7. 2024Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents
  8. 2026Doc-REFRAG: Rethinking Multimodal Document Retrieval-Augmented Generation