# A Simple Cache Model for Image Recognition

arXiv: https://arxiv.org/abs/1805.08709 · HTML: https://mlsanity.com/paper/1805.08709/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 May 2018.

## Built on by (most cited first, 20 of 20)

- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling](https://mlsanity.com/paper/2111.03930/) (2021)
- [Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification](https://mlsanity.com/paper/2207.09519/) (2022)
- [Efficient Test-Time Adaptation of Vision-Language Models](https://mlsanity.com/paper/2403.18293/) (2024)
- [Personalized Federated Learning through Local Memorization](https://mlsanity.com/paper/2111.09360/) (2021)
- [kNN Prompting: Beyond-Context Learning with Calibration-Free Nearest Neighbor Inference](https://mlsanity.com/paper/2303.13824/) (2023)
- [Frozen Feature Augmentation for Few-Shot Image Classification](https://mlsanity.com/paper/2403.10519/) (2024)
- [Rethinking Nearest Neighbors for Visual Classification](https://mlsanity.com/paper/2112.08459/) (2021)
- [Federated Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2302.12211/) (2023)
- [Point-Cache: Test-time Dynamic and Hierarchical Cache for Robust and Generalizable Point Cloud Analysis](https://mlsanity.com/paper/2503.12150/) (2025)
- [FIANCEE: Faster Inference of Adversarial Networks via Conditional Early Exits](https://mlsanity.com/paper/2304.10306/) (2023)
- [Compositional Caching for Training-free Open-vocabulary Attribute Detection](https://mlsanity.com/paper/2503.19145/) (2025)
- [Representation Memorization for Fast Learning New Knowledge without Forgetting](https://mlsanity.com/paper/2108.12596/) (2021)
- [Can deep learning match the efficiency of human visual long-term memory in storing object details?](https://mlsanity.com/paper/2204.13061/) (2022)
- [Revisiting k-NN for Fine-tuning Pre-trained Language Models](https://mlsanity.com/paper/2304.09058/) (2023)
- [Noise-Tolerant Few-Shot Unsupervised Adapter for Vision-Language Models](https://mlsanity.com/paper/2309.14928/) (2023)
- [Heterogenous Memory Augmented Neural Networks](https://mlsanity.com/paper/2310.10909/) (2023)
- [Learning to Adapt Category Consistent Meta-Feature of CLIP for Few-Shot Classification](https://mlsanity.com/paper/2407.05647/) (2024)
- [Advancing Cache-Based Few-Shot Classification via Patch-Driven Relational Gated Graph Attention](https://mlsanity.com/paper/2512.12498/) (2025)
- [Adapting Point Cloud Analysis via Multimodal Bayesian Distribution Learning](https://mlsanity.com/paper/2603.22070/) (2026)

## Abstract (arXiv)

Training large-scale image recognition models is computationally expensive. This raises the question of whether there might be simple ways to improve the test performance of an already trained model without having to re-train or fine-tune it with new data. Here, we show that, surprisingly, this is indeed possible. The key observation we make is that the layers of a deep network close to the output layer contain independent, easily extractable class-relevant information that is not contained in the output layer itself. We propose to extract this extra class-relevant information using a simple key-value cache memory to improve the classification performance of the model at test time. Our cache memory is directly inspired by a similar cache model previously proposed for language modeling (Grave et al., 2017). This cache component does not require any training or fine-tuning; it can be applied to any pre-trained model and, by properly setting only two hyper-parameters, leads to significant improvements in its classification performance. Improvements are observed across several architectures and datasets. In the cache component, using features extracted from layers close to the output (but not from the output layer itself) as keys leads to the largest improvements. Concatenating features from multiple layers to form keys can further improve performance over using single-layer features as keys. The cache component also has a regularizing effect, a simple consequence of which is that it substantially increases the robustness of models against adversarial attacks.