mlsanityData to 14 Oct 2026

Zero-shot Multimodal Document Retrieval via Cross-modal Question Generation

arXiv 2508.17079 ↗First posted 23 Aug 2025cs.IR cs.AIcited by 6

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Rapid advances in Multimodal Large Language Models (MLLMs) have expanded information retrieval beyond purely textual inputs, enabling retrieval from complex real world documents that combine text and visuals. However, most documents are private either owned by individuals or confined within corporate silos and current retrievers struggle when faced with unseen domains or languages. To address this gap, we introduce PREMIR, a simple yet effective framework that leverages the broad knowledge of an MLLM to generate cross modal pre questions (preQs) before retrieval. Unlike earlier multimodal retrievers that compare embeddings in a single vector space, PREMIR leverages preQs from multiple complementary modalities to expand the scope of matching to the token level. Experiments show that PREMIR achieves state of the art performance on out of distribution benchmarks, including closed domain and multilingual settings, outperforming strong baselines across all retrieval metrics. We confirm the contribution of each component through in depth ablation studies, and qualitative analyses of the generated preQs further highlight the model's robustness in real world settings.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 42 references that are in the map. First posted on arXiv 23 Aug 2025.

Builds on

28 of 42 references.

References · 28

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  6. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  8. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,261
  9. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  10. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
  11. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  12. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
16 more
  1. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  2. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 553
  3. 2023Large Language Models Are Zero-Shot Time Series Forecasterscited by 500
  4. 2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexitycited by 451
  5. 2024MinerU: An Open-Source Solution for Precise Document Content Extractioncited by 267
  6. 2021DocFormer: End-to-End Transformer for Document Understandingcited by 256
  7. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 245
  8. 2019Document Expansion by Query Predictioncited by 231
  9. 2022Unifying Vision, Text, and Layout for Universal Document Processingcited by 140
  10. 2021Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Surveycited by 130
  11. 2023Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluationscited by 97
  12. 2024Reducing hallucination in structured outputs via Retrieval-Augmented Generationcited by 93
  13. 2025ViDoRAG: Visual Document Retrieval-Augmented Generation via Dynamic Iterative Reasoning Agentscited by 66
  14. 2025REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmarkcited by 35
  15. 2024Leveraging Large Language Models for Medical Information Extraction and Query Generationcited by 8
  16. 2024CT2C-QA: Multimodal Question Answering over Chinese Text, Table and Chartcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025MemVerse: Multimodal Memory for Lifelong Learning Agentscited by 35
  2. 2025Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understandingcited by 9
  3. 2026V-Mem: Modality-Routed Retrieval for Long-Term Multimodal Agentic Memorycited by 4
  4. 2026Document-as-Image Representations Fall Short for Scientific Retrievalcited by 1
  5. 2026Dr. DocBench: A Comprehensive Benchmark for Expert-Level and Difficult Document Parsingcited by 0
  6. 2026MIDR: Enrichment-Augmented Indexing for Multimodal Document Retrievalcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025DocMMIR: A Framework for Document Multi-modal Information Retrieval
  2. 2026RETLLM: Training and Data-Free MLLMs for Multimodal Information Retrieval
  3. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text
  4. 2024MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs
  5. 2026Beyond Global Similarity: Towards Fine-Grained, Multi-Condition Multimodal Retrieval
  6. 2026CrossModalQA: A Cross-modal and Multi-hop Benchmark for Multimodal Retrieval-augmented Generation
  7. 2025VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understanding
  8. 2024PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrievers