mlsanityData to 7 Oct 2026

PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrievers

arXiv 2402.08327 ↗First posted 13 Feb 2024cs.CLcited by 42

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Large Multimodal Models (LMMs) excel in natural language and visual understanding but are challenged by exacting tasks such as Knowledge-based Visual Question Answering (KB-VQA) which involve the retrieval of relevant information from document collections to use in shaping answers to questions. We present an extensive training and evaluation framework, M2KR, for KB-VQA. M2KR contains a collection of vision and language tasks which we have incorporated into a single suite of benchmark tasks for training and evaluating general-purpose multi-modal retrievers. We use M2KR to develop PreFLMR, a pre-trained version of the recently developed Fine-grained Late-interaction Multi-modal Retriever (FLMR) approach to KB-VQA, and we report new state-of-the-art results across a range of tasks. We also present investigations into the scaling behaviors of PreFLMR intended to be useful in future developments in general-purpose multi-modal retrievers.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 65 references that are in the map. First posted on arXiv 13 Feb 2024.

Builds on

40 of 65 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Retrieval-based Knowledge Augmented Vision Language Pre-training27 Apr
  2. 2021FILIP: Fine-grained Interactive Language-Image Pre-Training9 Nov

Other references · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2017Billion-scale similarity search with GPUscited by 2,477
  10. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  12. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
26 more
  1. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  2. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  3. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  4. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  5. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  6. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  7. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  8. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  9. 2020Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrievalcited by 305
  10. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  11. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 203
  12. 2018Large Scale Fine-Grained Categorization and Domain-Specific Transfer Learningcited by 174
  13. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 165
  14. 2020A Theory of Usable Information Under Computational Constraintscited by 153
  15. 2022Revisiting Neural Scaling Laws in Language and Visioncited by 129
  16. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 125
  17. 2020KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQAcited by 124
  18. 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categoriescited by 105
  19. 2022PLAID: An Efficient Engine for Late Interaction Retrievalcited by 95
  20. 2022Retrieval Augmented Visual Question Answering with Outside Knowledgecited by 93
  21. 2021Multi-Modal Answer Validation for Knowledge-Based VQAcited by 78
  22. 2020Understanding tables with intermediate pre-trainingcited by 68
  23. 2022IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languagescited by 54
  24. 2021Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answeringcited by 50
  25. 2021Passage Retrieval for Outside-Knowledge Visual Question Answeringcited by 25
  26. 2022LaKo: Knowledge-driven Visual Question Answering via Late Knowledge-to-Text Injectioncited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 15; 2026: 19.

42 papers.

Sort:

By area:

  1. 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 166
  2. 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistantcited by 106
  3. 2024EchoSight: Advancing Visual-Language Models with Wiki Knowledgecited by 61
  4. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  5. 2024Automated Multi-level Preference for MLLMscited by 29
  6. 2025MMKB-RAG: A Multi-Modal Knowledge-Based Retrieval-Augmented Generation Frameworkcited by 18
  7. 2024Unified Generative and Discriminative Training for Multi-modal Large Language Modelscited by 15
  8. 2025Recurrence-Enhanced Vision-and-Language Transformers for Robust Multimodal Document Retrievalcited by 13
  9. 2024TIGeR: Unifying Text-to-Image Generation and Retrieval with Large Multimodal Modelscited by 12
  10. 2026According to Me: Long-Term Personalized Referential Memory QAcited by 12
  11. 2025OMGM: Orchestrate Multiple Granularities and Modalities for Efficient Multimodal Retrievalcited by 10
  12. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
30 more
  1. 2024Self-Bootstrapped Visual-Language Model for Knowledge Selection and Question Answeringcited by 5
  2. 2024MIRe: Enhancing Multimodal Queries Representation via Fusion-Free Modality Interaction for Multimodal Retrievalcited by 3
  3. 2025Everything Can Be Described in Words: A Simple Unified Multi-Modal Framework with Semantic and Temporal Alignmentcited by 3
  4. 2025AR-RAG: Autoregressive Retrieval Augmentation for Image Generationcited by 3
  5. 2025On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Surveycited by 3
  6. 2025Progressive Multimodal Search and Reasoning for Knowledge-Intensive Visual Question Answeringcited by 3
  7. 2025Recurrence Meets Transformers for Universal Multimodal Retrievalcited by 3
  8. 2025Towards Text-Image Interleaved Retrievalcited by 2
  9. 2025Enabling Collaborative Parametric Knowledge Calibration for Retrieval-Augmented Vision Question Answeringcited by 2
  10. 2025Breaking the Visual Shortcuts in Multimodal Knowledge-Based Visual Question Answeringcited by 2
  11. 2026V-Retrver: Evidence-Driven Agentic Reasoning for Universal Multimodal Retrievalcited by 2
  12. 2026Beyond Global Similarity: Towards Fine-Grained, Multi-Condition Multimodal Retrievalcited by 2
  13. 2025Taming a Retrieval Framework to Read Images in Humanlike Manner for Augmenting Generation of MLLMscited by 1
  14. 2026When RAG Hurts: Diagnosing and Mitigating Attention Distraction in Retrieval-Augmented LVLMscited by 1
  15. 2026REAL: Resolving Knowledge Conflicts in Knowledge-Intensive Visual Question Answering via Reasoning-Pivot Alignmentcited by 1
  16. 2026From Scenes to Elements: Multi-Granularity Evidence Retrieval for Verifiable Multimodal RAGcited by 1
  17. 2026Lost at the End: Primacy Bias in Multimodal Retrieval-Augmented Question Answeringcited by 1
  18. 2025MANTA: Cross-Modal Semantic Alignment and Information-Theoretic Optimization for Long-form Multimodal Understandingcited by 0
  19. 2025FoCLIP: A Feature-Space Misalignment Framework for CLIP-Based Image Manipulation and Detectioncited by 0
  20. 2026Relevance-aware Multi-context Contrastive Decoding for Retrieval-augmented Visual Question Answeringcited by 0
  21. 2026GEM: A Native Graph-based Index for Multi-Vector Retrievalcited by 0
  22. 2026BERAG: Bayesian Ensemble Retrieval-Augmented Generation for Knowledge-based Visual Question Answeringcited by 0
  23. 2026Overview of the EReL@MIR 2025 Multimodal Document Retrieval Challenge (Track 1)cited by 0
  24. 2026ColGraphRAG: Late-Interaction Evidence Retrieval for Multimodal GraphRAGcited by 0
  25. 2026UniHEAR: Unified Heterogeneous-Source Attentive Retrieval for Knowledge-Based Visual Question Answeringcited by 0
  26. 2026Bayesian Data Reweighting Improves Multimodal Retrieval for Knowledge-Based Visual Question Answeringcited by 0
  27. 2026ConceptFormer: Learning Adaptive Latent Concepts for Query-Document Alignment in Visual Document Retrievalcited by 0
  28. 2026Boosting Knowledge-based Visual Question Answering with Structured Context Reasoningcited by 0
  29. 2026Beyond Visual Similarity: Entity-Aligned Retrieval for Knowledge-Based Visual Question Answeringcited by 0
  30. 2026SAFE-G: Structure-aware Faithful Evidence-guided Generation for Knowledge-based Visual Question Answeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answering
  2. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey
  3. 2024mR2AG: Multimodal Retrieval-Reflection-Augmented Generation for Knowledge-Based VQA
  4. 2025MV-CoRe: Multimodal Visual-Conceptual Reasoning for Complex Visual Question Answering
  5. 2024Revisiting Multi-Modal LLM Evaluation
  6. 2024Visual Question Answering Instruction: Unlocking Multimodal Large Language Model To Domain-Specific Visual Multitasks
  7. 2026Enhancing Visual Question Answering with Multimodal LLMs via Chain-of-Question Guided Retrieval-Augmented Generation
  8. 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities