mlsanityData to 7 Oct 2026

Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification

arXiv 2211.11158 ↗First posted 21 Nov 2022cs.CV cs.CLcited by 310

Not on the map. Why?

Cited by 54 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Concept Bottleneck Models (CBM) are inherently interpretable models that factor model decisions into human-readable concepts. They allow people to easily understand why a model is failing, a critical feature for high-stakes applications. CBMs require manually specified concepts and often under-perform their black box counterparts, preventing their broad adoption. We address these shortcomings and are first to show how to construct high-performance CBMs without manual specification of similar accuracy to black box models. Our approach, Language Guided Bottlenecks (LaBo), leverages a language model, GPT-3, to define a large space of possible bottlenecks. Given a problem domain, LaBo uses GPT-3 to produce factual sentences about categories to form candidate concepts. LaBo efficiently searches possible bottlenecks through a novel submodular utility that promotes the selection of discriminative and diverse information. Ultimately, GPT-3's sentential concepts can be aligned to images using CLIP, to form a bottleneck layer. Experiments demonstrate that LaBo is a highly effective prior for concepts important to visual recognition. In the evaluation with 11 diverse datasets, LaBo bottlenecks excel at few-shot classification: they are 11.7% more accurate than black box linear probes at 1 shot and comparable with more data. Overall, LaBo demonstrates that inherently interpretable models can be widely applied at similar, or better, performance than black box approaches.

Key facts. Cited by 310 papers in the map (data through 7 Oct 2026). Builds on 55 of its 76 references that are in the map. First posted on arXiv 21 Nov 2022.

Builds on

55 of 76 references.

References · 55

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  6. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  10. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  11. 2016Matching Networks for One Shot Learningcited by 2,246
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
38 more
  1. 2013Describing Textures in the Wildcited by 2,110
  2. 2019Language Models as Knowledge Bases?cited by 1,922
  3. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  4. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  5. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  6. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  7. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  8. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  9. 2020Concept Bottleneck Modelscited by 1,022
  10. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  11. 2019How Can We Know What Language Models Know?cited by 953
  12. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  13. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  14. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  15. 2018This Looks Like That: Deep Learning for Interpretable Image Recognitioncited by 731
  16. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 703
  17. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  18. 2020Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?cited by 435
  19. 2022Visual Classification via Description from Large Language Modelscited by 331
  20. 2017Few-Shot Learning with Graph Neural Networkscited by 324
  21. 2020A Survey on Neural Network Interpretabilitycited by 291
  22. 2022Post-hoc Concept Bottleneck Modelscited by 289
  23. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 263
  24. 2020AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Promptscited by 246
  25. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  26. 2020Neural Prototype Trees for Interpretable Fine-grained Image Recognitioncited by 213
  27. 2018Multimodal Explanations: Justifying Decisions and Pointing to the Evidencecited by 179
  28. 2020Compositional Explanations of Neuronscited by 172
  29. 2020Attribute Prototype Network for Zero-Shot Learningcited by 160
  30. 2016Generating Visual Explanationscited by 156
  31. 2022Natural Language Descriptions of Deep Visual Featurescited by 136
  32. 2021Do Concept Bottleneck Models Learn as Intended?cited by 114
  33. 2021Open World Compositional Zero-Shot Learningcited by 100
  34. 2018Grounding Visual Explanationscited by 92
  35. 2021Can Language Models be Biomedical Knowledge Bases?cited by 83
  36. 2022Concept Bottleneck Model with Additional Unsupervised Conceptscited by 79
  37. 2022K-LITE: Learning Transferable Visual Models with External Knowledgecited by 79
  38. 2021Few-shot Image Classification: Just Use a Library of Pre-trained Feature Extractors and a Simple Classifiercited by 32

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 43; 2024: 100; 2025: 101; 2026: 66.

310 papers.

Sort:

By area:

  1. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  2. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  3. 2023In-Context Impersonation Reveals Large Language Models' Strengths and Biasescited by 154
  4. 2023Concept-based Explainable Artificial Intelligence: A Surveycited by 97
  5. 2024Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discoverycited by 91
  6. 2023Learning Concise and Descriptive Attributes for Visual Recognitioncited by 73
  7. 2024VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidancecited by 65
  8. 2024A Multimodal Automated Interpretability Agentcited by 62
  9. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  10. 2024LatentQA: Teaching LLMs to Decode Activations Into Natural Languagecited by 49
  11. 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 45
  12. 2024Incremental Residual Concept Bottleneck Modelscited by 43
38 more
  1. 2024Concept Bottleneck Large Language Modelscited by 41
  2. 2023If at First You Don't Succeed, Try, Try Again: Faithful Diffusion-based Text-to-Image Generation by Selectioncited by 40
  3. 2023Robust and Interpretable Medical Image Classifiers via Concept Bottleneck Modelscited by 40
  4. 2023Human Uncertainty in Concept-Based AI Systemscited by 39
  5. 2023ArGue: Attribute-Guided Prompt Tuning for Vision-Language Modelscited by 39
  6. 2024Mammo-CLIP: A Vision Language Foundation Model to Enhance Data Efficiency and Robustness in Mammographycited by 35
  7. 2023Collaborating Foundation Models for Domain Generalized Semantic Segmentationcited by 33
  8. 2024Energy-Based Concept Bottleneck Models: Unifying Prediction, Concept Intervention, and Probabilistic Interpretationscited by 33
  9. 2023Coarse-to-Fine Concept Bottleneck Modelscited by 31
  10. 2024Just Shift It: Test-Time Prototype Shifting for Zero-Shot Generalization with Vision-Language Modelscited by 31
  11. 2024Textual Knowledge Matters: Cross-Modality Co-Teaching for Generalized Visual Class Discoverycited by 30
  12. 2023Learning Interpretable Style Embeddings via Prompting LLMscited by 29
  13. 2023Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Modelscited by 28
  14. 2024Few-shot Adaptation of Multi-modal Foundation Models: A Surveycited by 28
  15. 2025Derm1M: A Million-scale Vision-Language Dataset Aligned with Clinical Ontology Knowledge for Dermatologycited by 28
  16. 2025Language Guided Concept Bottleneck Models for Interpretable Continual Learningcited by 28
  17. 2024Decomposing Disease Descriptions for Enhanced Pathology Detection: A Multi-Aspect Vision-Language Pre-training Frameworkcited by 27
  18. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 26
  19. 2023DesCo: Learning Object Recognition with Rich Language Descriptionscited by 25
  20. 2023SI-MIL: Taming Deep MIL for Self-Interpretability in Gigapixel Histopathologycited by 25
  21. 2023Training-free Zero-shot Composed Image Retrieval with Local Concept Rerankingcited by 23
  22. 2024XAI meets LLMs: A Survey of the Relation between Explainable AI and Large Language Modelscited by 23
  23. 2023Interpretability is in the Mind of the Beholder: A Causal Framework for Human-interpretable Representation Learningcited by 22
  24. 2023Concept Bottleneck with Visual Concept Filtering for Explainable Medical Image Classificationcited by 21
  25. 2023Interpretable-by-Design Text Understanding with Iteratively Generated Concept Bottleneckcited by 21
  26. 2024Semi-supervised Concept Bottleneck Modelscited by 21
  27. 2024Explaining Datasets in Words: Statistical Models with Natural Language Parameterscited by 21
  28. 2024Explain via Any Concept: Concept Bottleneck Model with Open Vocabulary Conceptscited by 20
  29. 2024PaCE: Parsimonious Concept Engineering for Large Language Modelscited by 19
  30. 2024LaMI-DETR: Open-Vocabulary Detection with Language Model Instructioncited by 19
  31. 2024Leveraging Cross-Modal Neighbor Representation for Improved CLIP Classificationcited by 18
  32. 2024Self-eXplainable AI for Medical Image Analysis: A Survey and New Outlookscited by 18
  33. 2025A Comprehensive Survey on Self-Interpretable Neural Networkscited by 18
  34. 2024Eliminating Information Leakage in Hard Concept Bottleneck Models with Supervised, Hierarchical Concept Learningcited by 17
  35. 2024Aligning Human Knowledge with Visual Concepts Towards Explainable Medical Image Classificationcited by 17
  36. 2024Concept Bottleneck Models Without Predefined Conceptscited by 17
  37. 2025Causally Reliable Concept Bottleneck Modelscited by 17
  38. 2024Pre-trained Vision-Language Models Learn Discoverable Visual Conceptscited by 16

50 most cited of 310 All 310 (JSON)