# PEEB: Part-based Image Classifiers with an Explainable and Editable Language Bottleneck

arXiv: https://arxiv.org/abs/2403.05297 · HTML: https://mlsanity.com/paper/2403.05297/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 65 references that are in the map.
- First posted on arXiv 8 Mar 2024.

## Builds on (most cited first, 30 of 39)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [This Looks Like That: Deep Learning for Interpretable Image Recognition](https://mlsanity.com/paper/1806.10574/) (2018)
- [Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly](https://mlsanity.com/paper/1707.00600/) (2017)
- [Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification](https://mlsanity.com/paper/2211.11158/) (2022)
- [Simple Open-Vocabulary Object Detection with Vision Transformers](https://mlsanity.com/paper/2205.06230/) (2022)
- [Neural Prototype Trees for Interpretable Fine-grained Image Recognition](https://mlsanity.com/paper/2012.02046/) (2020)
- [Benchmarking Representation Learning for Natural World Image Collections](https://mlsanity.com/paper/2103.16483/) (2021)
- [Label-Embedding for Image Classification](https://mlsanity.com/paper/1503.08677/) (2015)
- [Attribute Prototype Network for Zero-Shot Learning](https://mlsanity.com/paper/2008.08290/) (2020)
- [TransFG: A Transformer Architecture for Fine-grained Recognition](https://mlsanity.com/paper/2103.07976/) (2021)
- [Deformable ProtoPNet: An Interpretable Image Classifier Using Deformable Prototypes](https://mlsanity.com/paper/2111.15000/) (2021)
- [End-to-End Object Detection with Adaptive Clustering Transformer](https://mlsanity.com/paper/2011.09315/) (2020)
- [PartImageNet: A Large, High-Quality Dataset of Parts](https://mlsanity.com/paper/2112.00933/) (2021)
- [Waffling around for Performance: Visual Classification with Random Words and Broad Concepts](https://mlsanity.com/paper/2306.07282/) (2023)
- [A Generative Adversarial Approach for Zero-Shot Learning from Noisy Texts](https://mlsanity.com/paper/1712.01381/) (2017)
- [Learning Concise and Descriptive Attributes for Visual Recognition](https://mlsanity.com/paper/2308.03685/) (2023)
- [ProtoPFormer: Concentrating on Prototypical Parts in Vision Transformers for Interpretable Image Recognition](https://mlsanity.com/paper/2208.10431/) (2022)
- [This Looks Like That, Because ... Explaining Prototypes for Interpretable Image Recognition](https://mlsanity.com/paper/2011.02863/) (2020)
- [The Unreasonable Effectiveness of Noisy Data for Fine-Grained Recognition](https://mlsanity.com/paper/1511.06789/) (2015)
- [Link the head to the "beak": Zero Shot Learning from Noisy Text Description at Part Precision](https://mlsanity.com/paper/1709.01148/) (2017)

## Built on by (most cited first, 7 of 7)

- [VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidance](https://mlsanity.com/paper/2408.01432/) (2024)
- [DCBM: Data-Efficient Visual Concept Bottleneck Models](https://mlsanity.com/paper/2412.11576/) (2024)
- [Improving Concept Alignment in Vision-Language Concept Bottleneck Models](https://mlsanity.com/paper/2405.01825/) (2024)
- [A Comprehensive Survey on the Risks and Limitations of Concept-based Models](https://mlsanity.com/paper/2506.04237/) (2025)
- [TAB: Transformer Attention Bottlenecks enable User Intervention and Debugging in Vision-Language Models](https://mlsanity.com/paper/2412.18675/) (2024)
- [COCO-Tree: Compositional Hierarchical Concept Trees for Enhanced Reasoning in Vision Language Models](https://mlsanity.com/paper/2510.11012/) (2025)
- [SuoiAI: Building a Dataset for Aquatic Invertebrates in Vietnam](https://mlsanity.com/paper/2504.15252/) (2025)

## Abstract (arXiv)

CLIP-based classifiers rely on the prompt containing a {class name} that is known to the text encoder. Therefore, they perform poorly on new classes or the classes whose names rarely appear on the Internet (e.g., scientific names of birds). For fine-grained classification, we propose PEEB - an explainable and editable classifier to (1) express the class name into a set of text descriptors that describe the visual parts of that class; and (2) match the embeddings of the detected parts to their textual descriptors in each class to compute a logit score for classification. In a zero-shot setting where the class names are unknown, PEEB outperforms CLIP by a huge margin (~10x in top-1 accuracy). Compared to part-based classifiers, PEEB is not only the state-of-the-art (SOTA) on the supervised-learning setting (88.80% and 92.20% accuracy on CUB-200 and Dogs-120, respectively) but also the first to enable users to edit the text descriptors to form a new classifier without any re-training. Compared to concept bottleneck models, PEEB is also the SOTA in both zero-shot and supervised-learning settings.