# Improving Image Recognition by Retrieving from Web-Scale Image-Text Data

arXiv: https://arxiv.org/abs/2304.05173 · HTML: https://mlsanity.com/paper/2304.05173/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 47 of its 56 references that are in the map.
- First posted on arXiv 11 Apr 2023.

## Placed in

- Retrieval-augmented generation / [Multimodal RAG](https://mlsanity.com/rag/multimodal-rag/)

## Builds on (most cited first, 30 of 47)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021)
- [Generalization through Memorization: Nearest Neighbor Language Models](https://mlsanity.com/paper/1911.00172/) (2019)
- [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://mlsanity.com/paper/2209.06794/) (2022)
- [Large-Scale Long-Tailed Recognition in an Open World](https://mlsanity.com/paper/1904.05160/) (2019)
- [FLAVA: A Foundational Language And Vision Alignment Model](https://mlsanity.com/paper/2112.04482/) (2021)
- [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014)
- [Long-tail learning via logit adjustment](https://mlsanity.com/paper/2007.07314/) (2020)
- [LiT: Zero-Shot Transfer with Locked-image text Tuning](https://mlsanity.com/paper/2111.07991/) (2021)
- [Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks](https://mlsanity.com/paper/2208.10442/) (2022)
- [BBN: Bilateral-Branch Network with Cumulative Learning for Long-Tailed Visual Recognition](https://mlsanity.com/paper/1912.02413/) (2019)
- [Long-tailed Recognition by Routing Diverse Distribution-Aware Experts](https://mlsanity.com/paper/2010.01809/) (2020)
- [WebVision Database: Visual Learning and Understanding from Web Data](https://mlsanity.com/paper/1708.02862/) (2017)
- [YFCC100M: The New Data in Multimedia Research](https://mlsanity.com/paper/1503.01817/) (2015)
- [Memorizing Transformers](https://mlsanity.com/paper/2203.08913/) (2022)

## Built on by (most cited first, 20 of 20)

- [Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurts](https://mlsanity.com/paper/2309.10019/) (2023)
- [Retrieval-Enhanced Contrastive Vision-Text Models](https://mlsanity.com/paper/2306.07196/) (2023)
- [How to Benchmark Vision Foundation Models for Semantic Segmentation?](https://mlsanity.com/paper/2404.12172/) (2024)
- [RAVEN: Multitask Retrieval Augmented Vision-Language Learning](https://mlsanity.com/paper/2406.19150/) (2024)
- [A Generative Approach for Wikipedia-Scale Visual Entity Recognition](https://mlsanity.com/paper/2403.02041/) (2024)
- [Learning from Neighbors: Category Extrapolation for Long-Tail Learning](https://mlsanity.com/paper/2410.15980/) (2024)
- [SCoRe: Submodular Combinatorial Representation Learning](https://mlsanity.com/paper/2310.00165/) (2023)
- [SimplyRetrieve: A Private and Lightweight Retrieval-Centric Generative AI Tool](https://mlsanity.com/paper/2308.03983/) (2023)
- [Automatic Labelling with Open-source LLMs using Dynamic Label Schema Integration](https://mlsanity.com/paper/2501.12332/) (2025)
- [LIFT+: Lightweight Fine-Tuning for Long-Tail Learning](https://mlsanity.com/paper/2504.13282/) (2025)
- [Cross-Modal Retrieval Meets Inference:Improving Zero-Shot Classification with Cross-Modal Retrieval](https://mlsanity.com/paper/2308.15273/) (2023)
- [A Statistical Framework for Data-dependent Retrieval-Augmented Models](https://mlsanity.com/paper/2408.15399/) (2024)
- [Optimizing Skin Lesion Classification via Multimodal Data and Auxiliary Task Integration](https://mlsanity.com/paper/2402.10454/) (2024)
- [The Solution for the CVPR2024 NICE Image Captioning Challenge](https://mlsanity.com/paper/2404.12739/) (2024)
- [Leveraging Data to Say No: Memory Augmented Plug-and-Play Selective Prediction](https://mlsanity.com/paper/2601.22570/) (2026)
- [DREAM: Domain-agnostic Reverse Engineering Attributes of Black-box Model](https://mlsanity.com/paper/2412.05842/) (2024)
- [Memory-Modular Classification: Learning to Generalize with Memory Replacement](https://mlsanity.com/paper/2504.06021/) (2025)
- [ReCCur: A Recursive Corner-Case Curation Framework for Robust Vision-Language Understanding in Open and Edge Scenarios](https://mlsanity.com/paper/2601.03011/) (2026)
- [Retrieval-Enhanced Real Estate Appraisal](https://mlsanity.com/paper/2603.12986/) (2026)
- [Gen2Balance: Generative Balancing for Long-Tailed Video Action Recognition](https://mlsanity.com/paper/2606.22416/) (2026)

## Abstract (arXiv)

Retrieval augmented models are becoming increasingly popular for computer vision tasks after their recent success in NLP problems. The goal is to enhance the recognition capabilities of the model by retrieving similar examples for the visual input from an external memory set. In this work, we introduce an attention-based memory module, which learns the importance of each retrieved example from the memory. Compared to existing approaches, our method removes the influence of the irrelevant retrieved examples, and retains those that are beneficial to the input query. We also thoroughly study various ways of constructing the memory dataset. Our experiments show the benefit of using a massive-scale memory dataset of 1B image-text pairs, and demonstrate the performance of different memory representations. We evaluate our method in three different classification tasks, namely long-tailed recognition, learning with noisy labels, and fine-grained classification, and show that it achieves state-of-the-art accuracies in ImageNet-LT, Places-LT and Webvision datasets.