# PACE: Posthoc Architecture-Agnostic Concept Extractor for Explaining CNNs

arXiv: https://arxiv.org/abs/2108.13828 · HTML: https://mlsanity.com/paper/2108.13828/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 37 references that are in the map.
- First posted on arXiv 31 Aug 2021.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks](https://mlsanity.com/paper/1710.11063/) (2017)
- [This Looks Like That: Deep Learning for Interpretable Image Recognition](https://mlsanity.com/paper/1806.10574/) (2018)
- [Interpretable Explanations of Black Boxes by Meaningful Perturbation](https://mlsanity.com/paper/1704.03296/) (2017)
- [Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks](https://mlsanity.com/paper/1910.01279/) (2019)
- [Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly](https://mlsanity.com/paper/1707.00600/) (2017)
- [Top-down Neural Attention by Excitation Backprop](https://mlsanity.com/paper/1608.00507/) (2016)
- [Understanding Deep Networks via Extremal Perturbations and Smooth Masks](https://mlsanity.com/paper/1910.08485/) (2019)
- [On Completeness-aware Concept-Based Explanations in Deep Neural Networks](https://mlsanity.com/paper/1910.07969/) (2019)
- [Deep Learning for Case-Based Reasoning through Prototypes: A Neural Network that Explains Its Predictions](https://mlsanity.com/paper/1710.04806/) (2017)
- [Full-Gradient Representation for Neural Network Visualization](https://mlsanity.com/paper/1905.00780/) (2019)
- [Synthesizing the preferred inputs for neurons in neural networks via deep generator networks](https://mlsanity.com/paper/1605.09304/) (2016)
- [Interpretable Image Recognition with Hierarchical Prototypes](https://mlsanity.com/paper/1906.10651/) (2019)
- [The Doctor Just Won't Accept That!](https://mlsanity.com/paper/1711.08037/) (2017)
- [MAIRE -- A Model-Agnostic Interpretable Rule Extraction Procedure for Explaining Classifiers](https://mlsanity.com/paper/2011.01506/) (2020)

## Built on by (most cited first, 12 of 12)

- [Multi-dimensional concept discovery (MCD): A unifying framework with completeness guarantees](https://mlsanity.com/paper/2301.11911/) (2023)
- [A survey on Concept-based Approaches For Model Improvement](https://mlsanity.com/paper/2403.14566/) (2024)
- [Sparse Subspace Clustering for Concept Discovery (SSCCD)](https://mlsanity.com/paper/2203.06043/) (2022)
- [Scalable Concept Extraction in Industry 4.0](https://mlsanity.com/paper/2306.03551/) (2023)
- [Explainable AI for clinical risk prediction: a survey of concepts, methods, and modalities](https://mlsanity.com/paper/2308.08407/) (2023)
- [SurroCBM: Concept Bottleneck Surrogate Models for Generative Post-hoc Explanation](https://mlsanity.com/paper/2310.07698/) (2023)
- [GIFT: A Framework Towards Global Interpretable Faithful Textual Explanations of Vision Classifiers](https://mlsanity.com/paper/2411.15605/) (2024)
- [Explainable Supervised Domain Adaptation](https://mlsanity.com/paper/2205.09943/) (2022)
- [Multi-level Neural Networks for high-dimensional parametric obstacle problems](https://mlsanity.com/paper/2504.05026/) (2025)
- [FICNN: A Framework for the Interpretation of Deep Convolutional Neural Networks](https://mlsanity.com/paper/2305.10121/) (2023)
- [Deep Model Interpretation with Limited Data : A Coreset-based Approach](https://mlsanity.com/paper/2410.00524/) (2024)
- [CENDRe: Concept Extraction with Natural Domain Representations](https://mlsanity.com/paper/2607.29621/) (2026)

## Abstract (arXiv)

Deep CNNs, though have achieved the state of the art performance in image classification tasks, remain a black-box to a human using them. There is a growing interest in explaining the working of these deep models to improve their trustworthiness. In this paper, we introduce a Posthoc Architecture-agnostic Concept Extractor (PACE) that automatically extracts smaller sub-regions of the image called concepts relevant to the black-box prediction. PACE tightly integrates the faithfulness of the explanatory framework to the black-box model. To the best of our knowledge, this is the first work that extracts class-specific discriminative concepts in a posthoc manner automatically. The PACE framework is used to generate explanations for two different CNN architectures trained for classifying the AWA2 and Imagenet-Birds datasets. Extensive human subject experiments are conducted to validate the human interpretability and consistency of the explanations extracted by PACE. The results from these experiments suggest that over 72% of the concepts extracted by PACE are human interpretable.