mlsanityData to 14 Oct 2026

Developing Visual Augmented Q&A System using Scalable Vision Embedding Retrieval & Late Interaction Re-ranker

arXiv 2507.12378 ↗First posted 16 Jul 2025cs.IR cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Traditional information extraction systems face challenges with text only language models as it does not consider infographics (visual elements of information) such as tables, charts, images etc. often used to convey complex information to readers. Multimodal LLM (MLLM) face challenges of finding needle in the haystack problem i.e., either longer context length or substantial number of documents as search space. Late interaction mechanism over visual language models has shown state of the art performance in retrieval-based vision augmented Q&A tasks. There are yet few challenges using it for RAG based multi-modal Q&A. Firstly, many popular and widely adopted vector databases do not support native multi-vector retrieval. Secondly, late interaction requires computation which inflates space footprint and can hinder enterprise adoption. Lastly, the current state of late interaction mechanism does not leverage the approximate neighbor search indexing methods for large speed ups in retrieval process. This paper explores a pragmatic approach to make vision retrieval process scalable and efficient without compromising on performance quality. We propose multi-step custom implementation utilizing widely adopted hybrid search (metadata & embedding) and state of the art late interaction re-ranker to retrieve best matching pages. Finally, MLLM are prompted as reader to generate answers from contextualized best matching pages. Through experiments, we observe that the proposed design is scalable (significant speed up) and stable (without degrading performance quality), hence can be used as production systems at enterprises.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 8 of its 14 references that are in the map. First posted on arXiv 16 Jul 2025.

Builds on

8 of 14 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks7 Oct
  2. 2024ColPali: Efficient Document Retrieval with Vision Language Models27 Jun

Other references · 6

  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  2. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
  3. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 861
  4. 2024PaliGemma: A versatile 3B VLM for transfercited by 696
  5. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 553
  6. 2021A Comprehensive Survey and Experimental Comparison of Graph-Based Approximate Nearest Neighbor Searchcited by 82

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025OMGM: Orchestrate Multiple Granularities and Modalities for Efficient Multimodal Retrieval
  2. 2026WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering
  3. 2024Enhanced Multimodal RAG-LLM for Accurate Visual Question Answering
  4. 2026Enhancing Visual Question Answering with Multimodal LLMs via Chain-of-Question Guided Retrieval-Augmented Generation
  5. 2024Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents
  6. 2025Visual-RAG: Benchmarking Text-to-Image Retrieval Augmented Generation for Visual Knowledge Intensive Queries
  7. 2025Comparison of Text-Based and Image-Based Retrieval in Multimodal Retrieval Augmented Generation Large Language Model Systems
  8. 2024EchoSight: Advancing Visual-Language Models with Wiki Knowledge