ViSAR: Training-Free Adaptive-k Retrieval for Visual Document Question Answering

arXiv 2609.02486 ↗First posted 2 Sep 2026cs.IR cs.AI cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

also in 1 more

Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

Document Visual Question Answering (DocVQA) often leverages Retrieval-Augmented Generation (RAG), where late-interaction encoders are commonly used to identify document pages relevant to a user query, before answer generation by a Large Vision-Language Model (LVLM). Existing approaches typically retrieve a fixed top-k number of pages regardless of query complexity, which increases LVLM latency and may degrade answer accuracy. We introduce ViSAR (Visual Semantic Activation Retrieval), a training-free adaptive-k retrieval method for late-interaction visual document retrieval. ViSAR operates directly in the embedding space to construct a query-conditioned page-level similarity matrix that highlights query-relevant semantics and dynamically determines the number of pages to retrieve. Across multiple encoders and LVLMs, ViSAR retrieves compact, query-adapted page sets that reduce RAG latency by up to 58.7%, while maintaining or improving answer accuracy compared with fixed top-k and adaptive retrieval heuristics. Furthermore, we show that the similarity matrix structure correlates with answer accuracy, suggesting future directions for retrieval quality-aware document understanding.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 43 references that are in the map. First posted on arXiv 2 Sep 2026.

Builds on

39 of 43 references.

Closest ancestors · 3

Same direction, newest first

Other references · 36

2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
2025Qwen3-VL Technical Reportcited by 2,534
2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
2024A Survey on LLM-as-a-Judgecited by 1,384
2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexitycited by 437
2021InfographicVQAcited by 430
2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 237

24 more

2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 174
2021SPLADE v2: Sparse Lexical and Expansion Model for Information Retrievalcited by 126
2024M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understandingcited by 110
2022Hierarchical multimodal transformers for Multi-Page DocVQAcited by 102
2024ScreenAI: A Vision-Language Model for UI and Infographics Understandingcited by 97
2022PLAID: An Efficient Engine for Late Interaction Retrievalcited by 95
2025MDocAgent: A Multi-Modal Multi-Agent Framework for Document Understandingcited by 66
2019Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrievalcited by 65
2025VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documentscited by 57
2025No Free Labels: Limitations of LLM-as-a-Judge Without Human Groundingcited by 50
2024VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generationcited by 40
2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlookcited by 32
2021Pseudo-Relevance Feedback for Multiple Representation Dense Retrievalcited by 31
2025SimpleDoc: Multi-Modal Document Understanding with Dual-Cue Page Retrieval and Iterative Refinementcited by 27
2026ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarioscited by 22
2025Efficient Context Selection for Long-Context QA: No Tuning, No Iteration, Just Adaptive-kcited by 19
2025ModernVBERT: Towards Smaller Visual Document Retrieverscited by 19
2023SLIM: Sparsified Late Interaction for Multi-Vector Retrieval with Inverted Indexescited by 12
2025MHier-RAG: Multi-Modal RAG for Visual-Rich Document Question-Answering via Hierarchical and Multi-Granularity Reasoningcited by 11
2024An Analysis on Matching Mechanisms and Token Pruning for Late-interaction Modelscited by 7
2025Cluster-based Adaptive Retrieval: Dynamic Context Selection for RAG Applicationscited by 5
2025Survey on Question Answering over Visually Rich Documents: Methods, Challenges, and Trendscited by 4
2025Incorporating Token Importance in Multi-Vector Retrievalcited by 3
2025Enhancing Retrieval-Augmented Generation with Topic-Enriched Embeddings: A Hybrid Approach Integrating Traditional NLP Techniquescited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way