mlsanityData to 14 Oct 2026

Cross-Modal Retrieval Augmentation for Multi-Modal Classification

arXiv 2104.08108 ↗First posted 16 Apr 2021cs.CV cs.CLcited by 20

Not on the map. Why?

Abstract (arXiv)

Recent advances in using retrieval components over external knowledge sources have shown impressive results for a variety of downstream tasks in natural language processing. Here, we explore the use of unstructured external knowledge sources of images and their corresponding captions for improving visual question answering (VQA). First, we train a novel alignment model for embedding images and captions in the same space, which achieves substantial improvement in performance on image-caption retrieval w.r.t. similar methods. Second, we show that retrieval-augmented multi-modal transformers using the trained alignment model improve results on VQA over strong baselines. We further conduct extensive experiments to establish the promise of this approach, and examine novel applications for inference time such as hot-swapping indices.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 36 of its 57 references that are in the map. First posted on arXiv 16 Apr 2021.

Builds on

36 of 57 references.

References · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  7. 2017Billion-scale similarity search with GPUscited by 2,497
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  9. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  10. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  11. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  12. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
24 more
  1. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  2. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  3. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  4. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  5. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  6. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 692
  7. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  8. 2018Stacked Cross Attention for Image-Text Matchingcited by 416
  9. 2019Visual Semantic Reasoning for Image-Text Matchingcited by 185
  10. 2016FVQA: Fact-based Visual Question Answeringcited by 181
  11. 2020In Defense of Grid Features for Visual Question Answeringcited by 155
  12. 2019Supervised Multimodal Bitransformers for Classifying Images and Textcited by 130
  13. 2019CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrievalcited by 110
  14. 2015Explicit Knowledge-based Reasoning for Visual Question Answeringcited by 85
  15. 2017Learning Semantic Concepts and Order for Image and Sentence Matchingcited by 62
  16. 2018Pythia v0.1: the Winning Entry to the VQA Challenge 2018cited by 62
  17. 2020Augmenting Transformers with KNN-Based Composite Memory for Dialoguecited by 53
  18. 2020Fine-grained Visual Textual Alignment for Cross-Modal Retrieval using Transformer Encoderscited by 47
  19. 2020Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledgecited by 37
  20. 2018Straight to the Facts: Learning Knowledge Base Retrieval for Factual Visual Question Answeringcited by 30
  21. 2020Are we pretraining it right? Digging deeper into visio-linguistic pretrainingcited by 25
  22. 2015Exploring Nearest Neighbor Approaches for Image Captioningcited by 19
  23. 2020Transformer Reasoning Network for Image-Text Matching and Retrievalcited by 19
  24. 2020MoVie: Revisiting Modulated Convolutions for Visual Counting and Beyondcited by 11

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 4; 2023: 6; 2024: 6; 2025: 3.

20 papers.

Sort:

By area:

  1. 2022Retrieval-Augmented Multimodal Language Modelingcited by 125
  2. 2024StructRAG: Boosting Knowledge Intensive Reasoning of LLMs via Inference-time Hybrid Information Structurizationcited by 71
  3. 2022Retrieval-Enhanced Machine Learningcited by 47
  4. 2021An Image is Worth More Than a Thousand Words: Towards Disentanglement in the Wildcited by 30
  5. 2023Information Retrieval Meets Large Language Models: A Strategic Report from Chinese IR Communitycited by 29
  6. 2022Learning Multimodal Data Augmentation in Feature Spacecited by 18
  7. 2025Revisiting Visual Understanding in Multimodal Reasoning through a Lens of Image Perturbationcited by 14
  8. 2023X-TRA: Improving Chest X-ray Tasks with Cross-Modal Retrieval Augmentationcited by 10
  9. 2023Efficient Token-Guided Image-Text Retrieval with Consistent Multimodal Contrastive Trainingcited by 10
  10. 2024RAVEN: Multitask Retrieval Augmented Vision-Language Learningcited by 10
  11. 2022LoopITR: Combining Dual and Cross Encoder Architectures for Image-Text Retrievalcited by 9
  12. 2023Offline Reinforcement Learning with Imbalanced Datasetscited by 6
8 more
  1. 2024Multimodal Unsupervised Domain Generalization by Retrieving Across the Modality Gapcited by 3
  2. 2024Retrieval Augmented Recipe Generationcited by 3
  3. 2023Borrowing Human Senses: Comment-Aware Self-Training for Social Media Multimodal Classificationcited by 2
  4. 2024Harnessing Shared Relations via Multimodal Mixup Contrastive Learning for Multimodal Classificationcited by 2
  5. 2025MIDAS: Misalignment-based Data Augmentation Strategy for Imbalanced Multimodal Learningcited by 2
  6. 2025Augmented Vision-Language Models: A Systematic Reviewcited by 1
  7. 2023Retrieval-Augmented Classification with Decoupled Representationcited by 0
  8. 2024Retrieval Augmented Cross-Modal Tag Recommendation in Software Q&A Sitescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering
  2. 2026CrossModalQA: A Cross-modal and Multi-hop Benchmark for Multimodal Retrieval-augmented Generation
  3. 2024SnapNTell: Enhancing Entity-Centric Visual Question Answering with Retrieval Augmented Multimodal LLM
  4. 2025Open-Ended and Knowledge-Intensive Video Question Answering ↗
  5. 2025Multimodal Representation Alignment for Cross-modal Information Retrieval
  6. 2025Everything Can Be Described in Words: A Simple Unified Multi-Modal Framework with Semantic and Temporal Alignment
  7. 2025Fine-Grained Knowledge Structuring and Retrieval for Visual Question Answering
  8. 2024IIU: Independent Inference Units for Knowledge-based Visual Question Answering