# Meaningfully Debugging Model Mistakes using Conceptual Counterfactual Explanations

arXiv: https://arxiv.org/abs/2106.12723 · HTML: https://mlsanity.com/paper/2106.12723/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 55 references that are in the map.
- First posted on arXiv 24 Jun 2021.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR](https://mlsanity.com/paper/1711.00399/) (2017)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks](https://mlsanity.com/paper/1710.11063/) (2017)
- [Learning under Concept Drift: A Review](https://mlsanity.com/paper/2004.05785/) (2020)
- [Dynabench: Rethinking Benchmarking in NLP](https://mlsanity.com/paper/2104.14337/) (2021)
- [Detecting and Correcting for Label Shift with Black Box Predictors](https://mlsanity.com/paper/1802.03916/) (2018)
- [Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming](https://mlsanity.com/paper/1907.07484/) (2019)
- [Counterfactual Visual Explanations](https://mlsanity.com/paper/1904.07451/) (2019)
- [Explanations based on the Missing: Towards Contrastive Explanations with Pertinent Negatives](https://mlsanity.com/paper/1802.07623/) (2018)
- [Multiaccuracy: Black-Box Post-Processing for Fairness in Classification](https://mlsanity.com/paper/1805.12317/) (2018)
- [The What-If Tool: Interactive Probing of Machine Learning Models](https://mlsanity.com/paper/1907.04135/) (2019)
- [Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Dataset](https://mlsanity.com/paper/2104.09957/) (2021)
- [Semidefinite relaxations for certifying robustness to adversarial examples](https://mlsanity.com/paper/1811.01057/) (2018)
- [Gradio: Hassle-Free Sharing and Testing of ML Models in the Wild](https://mlsanity.com/paper/1906.02569/) (2019)
- [CheXclusion: Fairness gaps in deep chest X-ray classifiers](https://mlsanity.com/paper/2003.00827/) (2020)
- [A Patient-Centric Dataset of Images and Metadata for Identifying Melanomas Using Clinical Context](https://mlsanity.com/paper/2008.07360/) (2020)
- [Preserving Causal Constraints in Counterfactual Explanations for Machine Learning Classifiers](https://mlsanity.com/paper/1912.03277/) (2019)
- [Net2Vec: Quantifying and Explaining how Concepts are Encoded by Filters in Deep Neural Networks](https://mlsanity.com/paper/1801.03454/) (2018)
- [Don't Judge an Object by Its Context: Learning to Overcome Contextual Bias](https://mlsanity.com/paper/2001.03152/) (2020)

## Built on by (most cited first, 30 of 53)

- [Post-hoc Concept Bottleneck Models](https://mlsanity.com/paper/2205.15480/) (2022)
- [Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks](https://mlsanity.com/paper/2207.13243/) (2022)
- [Black-Box Access is Insufficient for Rigorous AI Audits](https://mlsanity.com/paper/2401.14446/) (2024)
- [Probabilistic Concept Bottleneck Models](https://mlsanity.com/paper/2306.01574/) (2023)
- [Distilling Model Failures as Directions in Latent Space](https://mlsanity.com/paper/2206.14754/) (2022)
- [Discover and Cure: Concept-aware Mitigation of Spurious Correlation](https://mlsanity.com/paper/2305.00650/) (2023)
- [TIBET: Identifying and Evaluating Biases in Text-to-Image Generative Models](https://mlsanity.com/paper/2312.01261/) (2023)
- [Discovering Bugs in Vision Models using Off-the-shelf Image Generation and Captioning](https://mlsanity.com/paper/2208.08831/) (2022)
- [SkinCon: A skin disease dataset densely annotated by domain experts for fine-grained model debugging and analysis](https://mlsanity.com/paper/2302.00785/) (2023)
- [Incremental Residual Concept Bottleneck Models](https://mlsanity.com/paper/2404.08978/) (2024)
- [MM-SpuBench: Towards Better Understanding of Spurious Biases in Multimodal LLMs](https://mlsanity.com/paper/2406.17126/) (2024)
- [ModelDiff: A Framework for Comparing Learning Algorithms](https://mlsanity.com/paper/2211.12491/) (2022)
- [Learning to Intervene on Concept Bottlenecks](https://mlsanity.com/paper/2308.13453/) (2023)
- [Beyond Concept Bottleneck Models: How to Make Black Boxes Intervenable?](https://mlsanity.com/paper/2401.13544/) (2024)
- [Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlation](https://mlsanity.com/paper/2307.07907/) (2023)
- [Concept-Based Explanations in Computer Vision: Where Are We and Where Could We Go?](https://mlsanity.com/paper/2409.13456/) (2024)
- [Do Large Code Models Understand Programming Concepts? Counterfactual Analysis for Code Predicates](https://mlsanity.com/paper/2402.05980/) (2024)
- [From Neural Activations to Concepts: A Survey on Explaining Concepts in Neural Networks](https://mlsanity.com/paper/2310.11884/) (2023)
- [Evaluating the Stability of Semantic Concept Representations in CNNs for Robust Explainability](https://mlsanity.com/paper/2304.14864/) (2023)
- [QualEval: Qualitative Evaluation for Model Improvement](https://mlsanity.com/paper/2311.02807/) (2023)
- [Invariant Learning via Diffusion Dreamed Distribution Shifts](https://mlsanity.com/paper/2211.10370/) (2022)
- [Learning Robust Classifiers with Self-Guided Spurious Correlation Mitigation](https://mlsanity.com/paper/2405.03649/) (2024)
- [Adaptive Concept Bottleneck for Foundation Models Under Distribution Shifts](https://mlsanity.com/paper/2412.14097/) (2024)
- [Investigating the Quality of DermaMNIST and Fitzpatrick17k Dermatological Image Datasets](https://mlsanity.com/paper/2401.14497/) (2024)
- [Counterfactual Concept Bottleneck Models](https://mlsanity.com/paper/2402.01408/) (2024)
- [Towards Compositionality in Concept Learning](https://mlsanity.com/paper/2406.18534/) (2024)
- [Constructing Concept-based Models to Mitigate Spurious Correlations with Minimal Human Effort](https://mlsanity.com/paper/2407.08947/) (2024)
- [Selective Concept Models: Permitting Stakeholder Customisation at Test-Time](https://mlsanity.com/paper/2306.08424/) (2023)
- [Interventional Probing in High Dimensions: An NLI Case Study](https://mlsanity.com/paper/2304.10346/) (2023)
- [Diffexplainer: Towards Cross-modal Global Explanations with Diffusion Models](https://mlsanity.com/paper/2404.02618/) (2024)

## Abstract (arXiv)

Understanding and explaining the mistakes made by trained models is critical to many machine learning objectives, such as improving robustness, addressing concept drift, and mitigating biases. However, this is often an ad hoc process that involves manually looking at the model's mistakes on many test samples and guessing at the underlying reasons for those incorrect predictions. In this paper, we propose a systematic approach, conceptual counterfactual explanations (CCE), that explains why a classifier makes a mistake on a particular test sample(s) in terms of human-understandable concepts (e.g. this zebra is misclassified as a dog because of faint stripes). We base CCE on two prior ideas: counterfactual explanations and concept activation vectors, and validate our approach on well-known pretrained models, showing that it explains the models' mistakes meaningfully. In addition, for new models trained on data with spurious correlations, CCE accurately identifies the spurious correlation as the cause of model mistakes from a single misclassified test sample. On two challenging medical applications, CCE generated useful insights, confirmed by clinicians, into biases and mistakes the model makes in real-world settings.