mlsanityData to 13 Oct 2026

PEEB: Part-based Image Classifiers with an Explainable and Editable Language Bottleneck

arXiv 2403.05297 ↗First posted 8 Mar 2024cs.CV cs.AI cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

CLIP-based classifiers rely on the prompt containing a {class name} that is known to the text encoder. Therefore, they perform poorly on new classes or the classes whose names rarely appear on the Internet (e.g., scientific names of birds). For fine-grained classification, we propose PEEB - an explainable and editable classifier to (1) express the class name into a set of text descriptors that describe the visual parts of that class; and (2) match the embeddings of the detected parts to their textual descriptors in each class to compute a logit score for classification. In a zero-shot setting where the class names are unknown, PEEB outperforms CLIP by a huge margin (~10x in top-1 accuracy). Compared to part-based classifiers, PEEB is not only the state-of-the-art (SOTA) on the supervised-learning setting (88.80% and 92.20% accuracy on CUB-200 and Dogs-120, respectively) but also the first to enable users to edit the text descriptors to form a new classifier without any re-training. Compared to concept bottleneck models, PEEB is also the SOTA in both zero-shot and supervised-learning settings.

Key facts. Cited by 7 papers in the map (data through 13 Oct 2026). Builds on 39 of its 65 references that are in the map. First posted on arXiv 8 Mar 2024.

Builds on

39 of 65 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  3. 2020Language Models are Few-Shot Learnerscited by 32,320
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,223
  6. 2020End-to-End Object Detection with Transformerscited by 7,396
  7. 2016Densely Connected Convolutional Networkscited by 6,707
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,574
  9. 2016Gaussian Error Linear Units (GELUs)cited by 3,472
  10. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,362
  11. 2020Concept Bottleneck Modelscited by 1,027
  12. 2018This Looks Like That: Deep Learning for Interpretable Image Recognitioncited by 735
27 more
  1. 2017Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Uglycited by 538
  2. 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 311
  3. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 304
  4. 2020Neural Prototype Trees for Interpretable Fine-grained Image Recognitioncited by 214
  5. 2021Benchmarking Representation Learning for Natural World Image Collectionscited by 167
  6. 2015Label-Embedding for Image Classificationcited by 164
  7. 2020Attribute Prototype Network for Zero-Shot Learningcited by 160
  8. 2021TransFG: A Transformer Architecture for Fine-grained Recognitioncited by 131
  9. 2021Deformable ProtoPNet: An Interpretable Image Classifier Using Deformable Prototypescited by 120
  10. 2020End-to-End Object Detection with Adaptive Clustering Transformercited by 109
  11. 2021PartImageNet: A Large, High-Quality Dataset of Partscited by 107
  12. 2023Waffling around for Performance: Visual Classification with Random Words and Broad Conceptscited by 100
  13. 2017A Generative Adversarial Approach for Zero-Shot Learning from Noisy Textscited by 96
  14. 2023Learning Concise and Descriptive Attributes for Visual Recognitioncited by 73
  15. 2022ProtoPFormer: Concentrating on Prototypical Parts in Vision Transformers for Interpretable Image Recognitioncited by 63
  16. 2020This Looks Like That, Because ... Explaining Prototypes for Interpretable Image Recognitioncited by 50
  17. 2015The Unreasonable Effectiveness of Noisy Data for Fine-Grained Recognitioncited by 48
  18. 2017Link the head to the "beak": Zero Shot Learning from Noisy Text Description at Part Precisioncited by 46
  19. 2022Visual correspondence-based explanations improve AI robustness and human-AI team accuracycited by 38
  20. 2023Coarse-to-Fine Concept Bottleneck Modelscited by 31
  21. 2019CANZSL: Cycle-Consistent Adversarial Networks for Zero-Shot Learning from Natural Languagecited by 23
  22. 2020ZEST: Zero-shot Learning from Text Descriptions using Textual Similarity and Visual Summarizationcited by 22
  23. 2022SR-GNN: Spatial Relation-aware Graph Neural Network for Fine-Grained Image Categorizationcited by 21
  24. 2023Follow-Up Differential Descriptions: Language Models Resolve Ambiguities for Image Classificationcited by 13
  25. 2023LLMs as Visual Explainers: Advancing Image Classification with Evolving Visual Descriptionscited by 11
  26. 2022Zero-Shot Classification by Logical Reasoning on Natural Language Explanationscited by 4
  27. 2021Zero-shot Learning with Class Description Regularizationcited by 2

Built on by

Cited by year

Citing papers per year: 2024: 4; 2025: 3.

7 papers.

  1. 2024VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidancecited by 65
  2. 2024DCBM: Data-Efficient Visual Concept Bottleneck Modelscited by 16
  3. 2024Improving Concept Alignment in Vision-Language Concept Bottleneck Modelscited by 4
  4. 2025A Comprehensive Survey on the Risks and Limitations of Concept-based Modelscited by 4
  5. 2024TAB: Transformer Attention Bottlenecks enable User Intervention and Debugging in Vision-Language Modelscited by 2
  6. 2025COCO-Tree: Compositional Hierarchical Concept Trees for Enhanced Reasoning in Vision Language Modelscited by 1
  7. 2025SuoiAI: Building a Dataset for Aquatic Invertebrates in Vietnamcited by 0