mlsanityData to 7 Oct 2026

LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant

arXiv 2412.01720 ↗First posted 2 Dec 2024cs.CVcited by 106

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

With the rapid advancement of multimodal information retrieval, increasingly complex retrieval tasks have emerged. Existing methods predominately rely on task-specific fine-tuning of vision-language models, often those trained with image-text contrastive learning. In this paper, we explore the possibility of re-purposing generative Large Multimodal Models (LMMs) for retrieval. This approach enables unifying all retrieval tasks under the same formulation and, more importantly, allows for extrapolation towards unseen retrieval tasks without additional training. Our contributions can be summarised in the following aspects: (i) We introduce LamRA, a versatile framework designed to empower LMMs with sophisticated retrieval and reranking capabilities. (ii) For retrieval, we adopt a two-stage training strategy comprising language-only pre-training and multimodal instruction tuning to progressively enhance LMM's retrieval performance. (iii) For reranking, we employ joint training for both pointwise and listwise reranking, offering two distinct ways to further boost the retrieval performance. (iv) Extensive experimental results underscore the efficacy of our method in handling more than ten retrieval tasks, demonstrating robust performance in both supervised and zero-shot settings, including scenarios involving previously unseen retrieval tasks.

Key facts. Cited by 106 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 58 of its 63 references that are in the map. First posted on arXiv 2 Dec 2024.

Builds on

58 of 63 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs4 Nov
  2. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks7 Oct
  3. 2024EchoSight: Advancing Visual-Language Models with Wiki Knowledge17 Jul
  4. 2024E5-V: Universal Embeddings with Multimodal Large Language Models17 Jul

Other references · 54

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  10. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2023Mistral 7Bcited by 3,065
38 more
  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  3. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  4. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  5. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  6. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  7. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  8. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  9. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  10. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  11. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  12. 2016Visual Dialogcited by 377
  13. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  14. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 298
  15. 2024UltraEdit: Instruction-based Fine-Grained Image Editing at Scalecited by 266
  16. 2021Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Modelscited by 244
  17. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 238
  18. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 203
  19. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  20. 2016Visual Storytellingcited by 176
  21. 2023Zero-Shot Composed Image Retrieval with Textual Inversioncited by 171
  22. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 170
  23. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 165
  24. 2024MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructionscited by 119
  25. 2023Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entitiescited by 112
  26. 2023DetGPT: Detect What You Need via Reasoningcited by 112
  27. 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categoriescited by 105
  28. 2024EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameterscited by 99
  29. 2020Smooth-AP: Smoothing the Path Towards Large-Scale Image Retrievalcited by 98
  30. 2021Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformerscited by 93
  31. 2023Scaling Sentence Embeddings with Large Language Modelscited by 83
  32. 2023Sentence-level Prompts Benefit Composed Image Retrievalcited by 59
  33. 2023Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answeringcited by 51
  34. 2023GeneCIS: A Benchmark for General Conditional Image Similaritycited by 46
  35. 2024Efficient Large Multi-modal Models via Visual Context Compressioncited by 46
  36. 2024Learn "No" to Say "Yes" Better: Improving Vision-Language Models via Negationscited by 45
  37. 2024PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrieverscited by 42
  38. 2024MatchTime: Towards Automatic Soccer Game Commentary Generationcited by 37

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 40; 2026: 66.

106 papers.

Sort:

By area:

  1. 2025VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documentscited by 110
  2. 2025Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMscited by 73
  3. 2025LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted Contrastive Learningcited by 54
  4. 2025UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learningcited by 41
  5. 2025UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddingscited by 37
  6. 2025Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrievalcited by 26
  7. 2025Universal Video Temporal Grounding with Generative Multi-modal Large Language Modelscited by 26
  8. 2025SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligencecited by 19
  9. 2025Object-centric Video Question Answering with Visual Grounding and Referringcited by 12
  10. 2025PUMA: Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learningcited by 11
  11. 2025U-MARVEL: Unveiling Key Factors for Universal Multimodal Retrieval via Embedding Learning with MLLMscited by 10
  12. 2026LLaVA-FA: Learning Fourier Approximation for Compressing Large Multimodal Modelscited by 10
38 more
  1. 2026PLUME: Latent Reasoning Based Universal Multimodal Embeddingcited by 10
  2. 2025ELIP: Enhanced Visual-Language Foundation Models for Image Retrievalcited by 9
  3. 2025Reasoning Guided Embeddings: Leveraging MLLM Reasoning for Improved Multimodal Retrievalcited by 9
  4. 2025Learning to Instruct for Visual Instruction Tuningcited by 8
  5. 2025ReMatch: Boosting Representation through Matching for Multimodal Retrievalcited by 8
  6. 2026Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Frameworkcited by 8
  7. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
  8. 2025Logic Unseen: Revealing the Logical Blindspots of Vision-Language Modelscited by 7
  9. 2025FreeRet: MLLMs as Training-Free Retrieverscited by 7
  10. 2026TRACE: Task-Adaptive Reasoning and Representation Learning for Universal Multimodal Retrievalcited by 7
  11. 2025WAVE: Learning Unified & Versatile Audio-Visual Embeddings with Multimodal LLMcited by 6
  12. 2026TSEmbed: Unlocking Task Scaling in Universal Multimodal Embeddingscited by 6
  13. 2025UniMoCo: Unified Modality Completion for Robust Multi-Modal Embeddingscited by 5
  14. 2025NEXT: Multi-Grained Mixture of Experts via Text-Modulation for Multi-Modal Object Re-Identificationcited by 5
  15. 2026LaME: Learning to Think in Latent Space for Multimodal Embedding via Information Bottleneckcited by 5
  16. 2026ELVA: Exploring Ranking-Driven Universal Multimodal Retrievalcited by 5
  17. 2025Bidirectional Likelihood Estimation with Multi-Modal Large Language Models for Text-Video Retrievalcited by 4
  18. 2026VeRVE: Versatile Retrieval for Videos via Unified Embeddingscited by 4
  19. 2026VidVec: Unlocking Video MLLM Embeddings for Video-Text Retrievalcited by 4
  20. 2026CREM: Compression-Driven Representation Enhancement for Multimodal Retrieval and Comprehensioncited by 4
  21. 2025Universal Retrieval for Multimodal Trajectory Modelingcited by 3
  22. 2025Progressive Multimodal Search and Reasoning for Knowledge-Intensive Visual Question Answeringcited by 3
  23. 2025Recurrence Meets Transformers for Universal Multimodal Retrievalcited by 3
  24. 2025Explore More, Learn Better: Parallel MLLM Embeddings under Mutual Information Minimizationcited by 3
  25. 2025Large Language Models Meet Extreme Multi-label Classification: Scaling and Multi-modal Frameworkcited by 3
  26. 2026ReCALL: Recalibrating Capability Degradation for MLLM-based Composed Image Retrievalcited by 3
  27. 2026Magic-MM-Embedding: Towards Visual-Token-Efficient Universal Multimodal Embedding with MLLMscited by 3
  28. 2026ARK: A Dual-Axis Multimodal Retrieval Benchmark along Reasoning and Knowledgecited by 3
  29. 2026Beyond the Grid: Layout-Informed Multi-Vector Retrieval with Parsed Visual Document Representationscited by 3
  30. 2026Think When Needed: Adaptive Reasoning-Driven Multimodal Embeddings with a Dual-LoRA Architecturecited by 3
  31. 2026TTE-Flash: Accelerating Reasoning-based Multimodal Representations via Think-Then-Embed Tokenscited by 3
  32. 2025Histopathology Multi-modal Embedding for Pathology Composed Retrievalcited by 2
  33. 2025On the Suitability of Reinforcement Fine-Tuning to Visual Taskscited by 2
  34. 2025X-CoT: Explainable Text-to-Video Retrieval via LLM-based Chain-of-Thought Reasoningcited by 2
  35. 2025VIRTUE: Visual-Interactive Text-Image Universal Embeddercited by 2
  36. 2025Breaking the Visual Shortcuts in Multimodal Knowledge-Based Visual Question Answeringcited by 2
  37. 2025M3DR: Towards Universal Multilingual Multimodal Document Retrievalcited by 2
  38. 2025Roles of MLLMs in Visually Rich Document Retrieval for RAG: A Surveycited by 2

50 most cited of 106 All 106 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Training
  2. 2022Retrieval-Augmented Multimodal Language Modeling
  3. 2024Generative Cross-Modal Retrieval: Memorizing Images in Multimodal Language Models for Retrieval and Beyond
  4. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation
  5. 2026Skim and Skip: Hierarchical Adaptive Inference for Efficient Multimodal Retrieval
  6. 2024TIGeR: Unifying Text-to-Image Generation and Retrieval with Large Multimodal Models
  7. 2025UniCoRN: Unified Commented Retrieval Network with LMMs
  8. 2025Re-ranking the Context for Multimodal Retrieval Augmented Generation