# Revisiting a kNN-based Image Classification System with High-capacity Storage

arXiv: https://arxiv.org/abs/2204.01186 · HTML: https://mlsanity.com/paper/2204.01186/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 62 references that are in the map.
- First posted on arXiv 3 Apr 2022.

## Builds on (most cited first, 30 of 45)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [Transformer Interpretability Beyond Attention Visualization](https://mlsanity.com/paper/2012.09838/) (2020)
- [CoAtNet: Marrying Convolution and Attention for All Data Sizes](https://mlsanity.com/paper/2106.04803/) (2021)
- [Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://mlsanity.com/paper/2103.14749/) (2021)

## Built on by (most cited first, 22 of 22)

- [Online Continual Learning Without the Storage Constraint](https://mlsanity.com/paper/2305.09253/) (2023)
- [Improving Image Recognition by Retrieving from Web-Scale Image-Text Data](https://mlsanity.com/paper/2304.05173/) (2023)
- [Rethinking model prototyping through the MedMNIST+ dataset collection](https://mlsanity.com/paper/2404.15786/) (2024)
- [Towards flexible perception with visual memory](https://mlsanity.com/paper/2408.08172/) (2024)
- [Retrieval-Enhanced Visual Prompt Learning for Few-shot Classification](https://mlsanity.com/paper/2306.02243/) (2023)
- [SimplyRetrieve: A Private and Lightweight Retrieval-Centric Generative AI Tool](https://mlsanity.com/paper/2308.03983/) (2023)
- [Integrating kNN with Foundation Models for Adaptable and Privacy-Aware Image Classification](https://mlsanity.com/paper/2402.12500/) (2024)
- [An Embedding is Worth a Thousand Noisy Labels](https://mlsanity.com/paper/2408.14358/) (2024)
- [Revisiting Relevance Feedback for CLIP-based Interactive Image Retrieval](https://mlsanity.com/paper/2404.16398/) (2024)
- [Context-Aware Multimodal Pretraining](https://mlsanity.com/paper/2411.15099/) (2024)
- [IsoCLIP: Decomposing CLIP Projectors for Efficient Intra-modal Alignment](https://mlsanity.com/paper/2603.19862/) (2026)
- [Learning Group Activity Features Through Person Attribute Prediction](https://mlsanity.com/paper/2403.02753/) (2024)
- [Privacy-preserving datasets by capturing feature distributions with Conditional VAEs](https://mlsanity.com/paper/2408.00639/) (2024)
- [Improving Image Clustering with Artifacts Attenuation via Inference-Time Attention Engineering](https://mlsanity.com/paper/2410.04801/) (2024)
- [PARMESAN: Parameter-Free Memory Search and Transduction for Dense Prediction Tasks](https://mlsanity.com/paper/2403.11743/) (2024)
- [Synthetic Similarity Search in Automotive Production](https://mlsanity.com/paper/2505.07256/) (2025)
- [Leveraging Data to Say No: Memory Augmented Plug-and-Play Selective Prediction](https://mlsanity.com/paper/2601.22570/) (2026)
- [On Storage Neural Network Augmented Approximate Nearest Neighbor Search](https://mlsanity.com/paper/2501.16375/) (2025)
- [Memory-Modular Classification: Learning to Generalize with Memory Replacement](https://mlsanity.com/paper/2504.06021/) (2025)
- [Separating Knowledge and Perception with Procedural Data](https://mlsanity.com/paper/2508.11697/) (2025)
- [Vertical Fusion: Condensing Internal Representations for Robust ViT Classification](https://mlsanity.com/paper/2607.10391/) (2026)
- [TDDN: Text-aligned Diffused DINO Network for Puzzle Understanding](https://mlsanity.com/paper/2609.07937/) (2026)

## Abstract (arXiv)

In existing image classification systems that use deep neural networks, the knowledge needed for image classification is implicitly stored in model parameters. If users want to update this knowledge, then they need to fine-tune the model parameters. Moreover, users cannot verify the validity of inference results or evaluate the contribution of knowledge to the results. In this paper, we investigate a system that stores knowledge for image classification, such as image feature maps, labels, and original images, not in model parameters but in external high-capacity storage. Our system refers to the storage like a database when classifying input images. To increase knowledge, our system updates the database instead of fine-tuning model parameters, which avoids catastrophic forgetting in incremental learning scenarios. We revisit a kNN (k-Nearest Neighbor) classifier and employ it in our system. By analyzing the neighborhood samples referred by the kNN algorithm, we can interpret how knowledge learned in the past is used for inference results. Our system achieves 79.8% top-1 accuracy on the ImageNet dataset without fine-tuning model parameters after pretraining, and 90.8% accuracy on the Split CIFAR-100 dataset in the task incremental learning setting.