mlsanityData to 7 Oct 2026

Learning Customized Visual Models with Retrieval-Augmented Knowledge

arXiv 2301.07094 ↗First posted 17 Jan 2023cs.CV cs.AI cs.CLcited by 58

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

Image-text contrastive learning models such as CLIP have demonstrated strong task transfer ability. The high generality and usability of these visual models is achieved via a web-scale data collection process to ensure broad concept coverage, followed by expensive pre-training to feed all the knowledge into model weights. Alternatively, we propose REACT, REtrieval-Augmented CusTomization, a framework to acquire the relevant web knowledge to build customized visual models for target domains. We retrieve the most relevant image-text pairs (~3% of CLIP pre-training data) from the web-scale database as external knowledge, and propose to customize the model by only training new modualized blocks while freezing all the original weights. The effectiveness of REACT is demonstrated via extensive experiments on classification, retrieval, detection and segmentation tasks, including zero, few, and full-shot settings. Particularly, on the zero-shot classification task, compared with CLIP, it achieves up to 5.4% improvement on ImageNet and 3.7% on the ELEVATER benchmark (20 datasets).

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 77 of its 96 references that are in the map. First posted on arXiv 17 Jan 2023.

Builds on

77 of 96 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Retrieval-Augmented Multimodal Language Modeling22 Nov
  2. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text6 Oct
  3. 2022Re-Imagen: Retrieval-Augmented Text-to-Image Generator29 Sep
  4. 2022Retrieval Augmented Classification for Long-Tail Visual Recognition22 Feb

Other references · 73

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
38 more
  1. 2017Billion-scale similarity search with GPUscited by 2,477
  2. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  3. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  4. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  5. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  6. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  7. 2019Natural Adversarial Examplescited by 1,401
  8. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  9. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  10. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  11. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  12. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  13. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  14. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  15. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  16. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  17. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 853
  18. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  19. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  20. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  21. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  22. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  23. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  24. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  25. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  26. 2021Extract Free Dense Labels from CLIPcited by 580
  27. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  28. 2018Do Better ImageNet Models Transfer Better?cited by 560
  29. 2017Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Uglycited by 536
  30. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  31. 2018Rotation Equivariant CNNs for Digital Pathologycited by 420
  32. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  33. 2019K-BERT: Enabling Language Representation with Knowledge Graphcited by 303
  34. 2022Masked Siamese Networks for Label-Efficient Learningcited by 301
  35. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  36. 2019Knowledge Enhanced Contextual Word Representationscited by 294
  37. 2015YFCC100M: The New Data in Multimedia Researchcited by 234
  38. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 21; 2025: 21; 2026: 3.

58 papers.

Sort:

By area:

  1. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  2. 2023Scaling Vision-Language Models with Sparse Mixture of Expertscited by 88
  3. 2023An Inverse Scaling Law for CLIP Trainingcited by 76
  4. 2024The Neglected Tails in Vision-Language Modelscited by 63
  5. 2023Text-guided Foundation Model Adaptation for Pathological Image Classificationcited by 47
  6. 2024ViTamin: Designing Scalable Vision Models in the Vision-Language Eracited by 39
  7. 2024In Defense of Lazy Visual Grounding for Open-Vocabulary Semantic Segmentationcited by 38
  8. 2025SpatialLLM: A Compound 3D-Informed Design towards Spatially-Intelligent Large Multimodal Modelscited by 37
  9. 2023Retrieval-Enhanced Contrastive Vision-Text Modelscited by 36
  10. 2025Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answeringcited by 33
  11. 2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlookcited by 32
  12. 2023Learning Hierarchical Prompt with Structured Linguistic Knowledge for Vision-Language Modelscited by 23
38 more
  1. 2024MoDE: CLIP Data Experts via Clusteringcited by 22
  2. 2024kNN-CLIP: Retrieval Enables Training-Free Segmentation on Continually Expanding Large Vocabulariescited by 20
  3. 2024Retrieval-augmented Few-shot Medical Image Segmentation with Foundation Modelscited by 20
  4. 2024The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Bettercited by 19
  5. 2024Few-Shot Recognition via Stage-Wise Retrieval-Augmented Finetuningcited by 15
  6. 2023Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Modelscited by 14
  7. 2025Patho-AgenticRAG: Towards Multimodal Agentic Retrieval-Augmented Generation for Pathology VLMs via Reinforcement Learningcited by 13
  8. 2023Neural Priming for Sample-Efficient Adaptationcited by 12
  9. 2023DeepSpeed-VisualChat: Multi-Round Multi-Image Interleave Chat via Multi-Modal Causal Attentioncited by 10
  10. 2024Understanding Retrieval-Augmented Task Adaptation for Vision-Language Modelscited by 10
  11. 2025Memory-QA: Answering Recall Questions Based on Multimodal Memoriescited by 10
  12. 2023NICE: CVPR 2023 Challenge on Zero-shot Image Captioningcited by 9
  13. 2024A Generative Approach for Wikipedia-Scale Visual Entity Recognitioncited by 9
  14. 2024Finding NeMo: Negative-mined Mosaic Augmentation for Referring Image Segmentationcited by 7
  15. 2024CaMML: Context-Aware Multimodal Learner for Large Modelscited by 6
  16. 2024Retrieval Augmented Generation for Dynamic Graph Modelingcited by 6
  17. 2023Benchmarking and Analyzing Generative Data for Visual Recognitioncited by 5
  18. 2023Meta Co-Training: Two Views are Better than Onecited by 5
  19. 2024Web-Scale Visual Entity Recognition: An LLM-Driven Data Approachcited by 5
  20. 2025Sign Language Translation using Frame and Event Stream: Benchmark Dataset and Algorithmscited by 4
  21. 2025TRiCo: Triadic Game-Theoretic Co-Training for Robust Semi-Supervised Learningcited by 4
  22. 2024Multimodal Unsupervised Domain Generalization by Retrieving Across the Modality Gapcited by 3
  23. 2026SARE: Sample-wise Adaptive Reasoning for Training-free Fine-grained Visual Recognitioncited by 3
  24. 2024HPT++: Hierarchically Prompting Vision-Language Models with Multi-Granularity Knowledge Generation and Improved Structure Modelingcited by 2
  25. 2024Enhancing Explainability in Multimodal Large Language Models Using Ontological Contextcited by 2
  26. 2024COBRA: COmBinatorial Retrieval Augmentation for Few-Shot Adaptationcited by 2
  27. 2025Enabling Collaborative Parametric Knowledge Calibration for Retrieval-Augmented Vision Question Answeringcited by 2
  28. 2025Active Learning via Vision-Language Model Adaptation with Open Datacited by 2
  29. 2025Enabling Validation for Robust Few-Shot Recognitioncited by 2
  30. 2025Scalable Disk-Based Approximate Nearest Neighbor Search with Page-Aligned Graphcited by 2
  31. 2023Annotating Ambiguous Images: General Annotation Strategy for High-Quality Data with Real-World Biomedical Validationcited by 1
  32. 2025Simple yet Effective Semi-supervised Knowledge Distillation from Vision-Language Models via Dual-Head Optimizationcited by 1
  33. 2025LLM-enhanced Action-aware Multi-modal Prompt Tuning for Image-Text Matchingcited by 1
  34. 2025MSGCoOp: Multiple Semantic-Guided Context Optimization for Few-Shot Learningcited by 1
  35. 2025Cohort-Aware Agents for Individualized Lung Cancer Risk Prediction Using a Retrieval-Augmented Model Selection Frameworkcited by 1
  36. 2025Memory-SAM: Human-Prompt-Free Tongue Segmentation via Retrieval-to-Promptcited by 1
  37. 2026Leveraging Data to Say No: Memory Augmented Plug-and-Play Selective Predictioncited by 1
  38. 2024Online Learning via Memory: Retrieval-Augmented Detector Adaptationcited by 0

50 most cited of 58 All 58 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Compositional Image Retrieval via Instruction-Aware Contrastive Learning
  2. 2026KIRA: Knowledge-Intensive Image Retrieval and Reasoning Architecture for Specialized Visual Domains
  3. 2025ImageRAG: Dynamic Image Retrieval for Reference-Guided Image Generation
  4. 2024Toward Automatic Relevance Judgment using Vision--Language Models for Image--Text Retrieval Evaluation
  5. 2023Enhancing CLIP with GPT-4: Harnessing Visual Descriptions as Prompts
  6. 2025CalibCLIP: Contextual Calibration of Dominant Semantics for Text-Driven Image Retrieval
  7. 2022Contrastive Language-Image Pre-Training with Knowledge Graphs
  8. 2024Fully Fine-tuned CLIP Models are Efficient Few-Shot Learners