# Visual Recognition with Deep Nearest Centroids

arXiv: https://arxiv.org/abs/2209.07383 · HTML: https://mlsanity.com/paper/2209.07383/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Builds on 88 of its 144 references that are in the map.
- First posted on arXiv 15 Sep 2022.

## Builds on (most cited first, 30 of 88)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)

## Built on by (most cited first, 30 of 59)

- [GMMSeg: Gaussian Mixture based Generative Semantic Segmentation Models](https://mlsanity.com/paper/2210.02025/) (2022)
- [Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinement](https://mlsanity.com/paper/2304.01195/) (2023)
- [E^2VPT: An Effective and Efficient Approach for Visual Prompt Tuning](https://mlsanity.com/paper/2307.13770/) (2023)
- [Image Segmentation in Foundation Model Era: A Survey](https://mlsanity.com/paper/2408.12957/) (2024)
- [TransFlow: Transformer as Flow Learner](https://mlsanity.com/paper/2304.11523/) (2023)
- [CLUSTSEG: Clustering for Universal Segmentation](https://mlsanity.com/paper/2305.02187/) (2023)
- [Facing the Elephant in the Room: Visual Prompt Tuning or Full Finetuning?](https://mlsanity.com/paper/2401.12902/) (2024)
- [A Comprehensive Survey on Self-Interpretable Neural Networks](https://mlsanity.com/paper/2501.15638/) (2025)
- [Learning Equivariant Segmentation with Instance-Unique Querying](https://mlsanity.com/paper/2210.00911/) (2022)
- [Visual Knowledge in the Big Model Era: Retrospect and Prospect](https://mlsanity.com/paper/2404.04308/) (2024)
- [M2PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning](https://mlsanity.com/paper/2409.15657/) (2024)
- [Kernel-Aware Graph Prompt Learning for Few-Shot Anomaly Detection](https://mlsanity.com/paper/2412.17619/) (2024)
- [Detect Everything with Few Examples](https://mlsanity.com/paper/2309.12969/) (2023)
- [Learning Clustering-based Prototypes for Compositional Zero-shot Learning](https://mlsanity.com/paper/2502.06501/) (2025)
- [Neural Clustering based Visual Representation Learning](https://mlsanity.com/paper/2403.17409/) (2024)
- [AMD: Automatic Multi-step Distillation of Large-scale Vision Models](https://mlsanity.com/paper/2407.04208/) (2024)
- [Interpretable Image Classification via Non-parametric Part Prototype Learning](https://mlsanity.com/paper/2503.10247/) (2025)
- [Re-Imagining Multimodal Instruction Tuning: A Representation View](https://mlsanity.com/paper/2503.00723/) (2025)
- [MoSEs: Uncertainty-Aware AI-Generated Text Detection via Mixture of Stylistics Experts with Conditional Thresholds](https://mlsanity.com/paper/2509.02499/) (2025)
- [ClusterFormer: Clustering As A Universal Visual Learner](https://mlsanity.com/paper/2309.13196/) (2023)
- [Clustering for Protein Representation Learning](https://mlsanity.com/paper/2404.00254/) (2024)
- [Learning Unknowns from Unknowns: Diversified Negative Prototypes Generator for Few-Shot Open-Set Recognition](https://mlsanity.com/paper/2408.13373/) (2024)
- [Image Translation as Diffusion Visual Programmers](https://mlsanity.com/paper/2401.09742/) (2024)
- [ProMotion: Prototypes As Motion Learners](https://mlsanity.com/paper/2406.04999/) (2024)
- [SSA-Seg: Semantic and Spatial Adaptive Pixel-level Classifier for Semantic Segmentation](https://mlsanity.com/paper/2405.06525/) (2024)
- [Prototypical Transformer as Unified Motion Learners](https://mlsanity.com/paper/2406.01559/) (2024)
- [Multi-Scale Grouped Prototypes for Interpretable Semantic Segmentation](https://mlsanity.com/paper/2409.09497/) (2024)
- [Cross- and Intra-image Prototypical Learning for Multi-label Disease Diagnosis and Interpretation](https://mlsanity.com/paper/2411.04607/) (2024)
- [Towards interpretable-by-design deep learning algorithms](https://mlsanity.com/paper/2311.11396/) (2023)
- [Hyperspherical Classification with Dynamic Label-to-Prototype Assignment](https://mlsanity.com/paper/2403.16937/) (2024)

## Abstract (arXiv)

We devise deep nearest centroids (DNC), a conceptually elegant yet surprisingly effective network for large-scale visual recognition, by revisiting Nearest Centroids, one of the most classic and simple classifiers. Current deep models learn the classifier in a fully parametric manner, ignoring the latent data structure and lacking simplicity and explainability. DNC instead conducts nonparametric, case-based reasoning; it utilizes sub-centroids of training samples to describe class distributions and clearly explains the classification as the proximity of test data and the class sub-centroids in the feature space. Due to the distance-based nature, the network output dimensionality is flexible, and all the learnable parameters are only for data embedding. That means all the knowledge learnt for ImageNet classification can be completely transferred for pixel recognition learning, under the "pre-training and fine-tuning" paradigm. Apart from its nested simplicity and intuitive decision-making mechanism, DNC can even possess ad-hoc explainability when the sub-centroids are selected as actual training images that humans can view and inspect. Compared with parametric counterparts, DNC performs better on image classification (CIFAR-10, ImageNet) and greatly boots pixel recognition (ADE20K, Cityscapes), with improved transparency and fewer learnable parameters, using various network architectures (ResNet, Swin) and segmentation models (FCN, DeepLabV3, Swin). We feel this work brings fundamental insights into related fields.