# Defining and Quantifying the Emergence of Sparse Concepts in DNNs

arXiv: https://arxiv.org/abs/2111.06206 · HTML: https://mlsanity.com/paper/2111.06206/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 95 references that are in the map.
- First posted on arXiv 11 Nov 2021.

## Builds on (most cited first, 30 of 54)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [Interpretable Explanations of Black Boxes by Meaningful Perturbation](https://mlsanity.com/paper/1704.03296/) (2017)
- [Captum: A unified and generic model interpretability library for PyTorch](https://mlsanity.com/paper/2009.07896/) (2020)
- [Understanding Neural Networks Through Deep Visualization](https://mlsanity.com/paper/1506.06579/) (2015)
- [Not Just a Black Box: Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1605.01713/) (2016)
- [Consistent Individualized Feature Attribution for Tree Ensembles](https://mlsanity.com/paper/1802.03888/) (2018)
- [Understanding Deep Networks via Extremal Perturbations and Smooth Masks](https://mlsanity.com/paper/1910.08485/) (2019)
- [Learning to Explain: An Information-Theoretic Perspective on Model Interpretation](https://mlsanity.com/paper/1802.07814/) (2018)
- [Real Time Image Saliency for Black Box Classifiers](https://mlsanity.com/paper/1705.07857/) (2017)
- [Distilling a Neural Network Into a Soft Decision Tree](https://mlsanity.com/paper/1711.09784/) (2017)
- [Object Detectors Emerge in Deep Scene CNNs](https://mlsanity.com/paper/1412.6856/) (2014)
- [Inverting Visual Representations with Convolutional Networks](https://mlsanity.com/paper/1506.02753/) (2015)
- [Abduction-Based Explanations for Machine Learning Models](https://mlsanity.com/paper/1811.10656/) (2018)
- [Improving KernelSHAP: Practical Shapley Value Estimation via Linear Regression](https://mlsanity.com/paper/2012.01536/) (2020)
- [The Shapley Taylor Interaction Index](https://mlsanity.com/paper/1902.05622/) (2019)
- [Explaining Deep Neural Networks with a Polynomial Time Algorithm for Shapley Values Approximation](https://mlsanity.com/paper/1903.10992/) (2019)
- [Asymmetric Shapley values: incorporating causal knowledge into model-agnostic explainability](https://mlsanity.com/paper/1910.06358/) (2019)

## Built on by (most cited first, 30 of 36)

- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [A Comprehensive Study of Knowledge Editing for Large Language Models](https://mlsanity.com/paper/2401.01286/) (2024)
- [Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2412.02104/) (2024)
- [Identifying Semantic Induction Heads to Understand In-Context Learning](https://mlsanity.com/paper/2402.13055/) (2024)
- [Explaining Generalization Power of a DNN Using Interactive Concepts](https://mlsanity.com/paper/2302.13091/) (2023)
- [HarsanyiNet: Computing Accurate Shapley Values in a Single Forward Propagation](https://mlsanity.com/paper/2304.01811/) (2023)
- [Defining and Extracting generalizable interaction primitives from DNNs](https://mlsanity.com/paper/2401.16318/) (2024)
- [Towards the Dynamics of a DNN Learning Symbolic Interactions](https://mlsanity.com/paper/2407.19198/) (2024)
- [Technical Note: Defining and Quantifying AND-OR Interactions for Faithful and Concise Explanation of DNNs](https://mlsanity.com/paper/2304.13312/) (2023)
- [Can the Inference Logic of Large Language Models be Disentangled into Symbolic Concepts?](https://mlsanity.com/paper/2304.01083/) (2023)
- [INTER: Mitigating Hallucination in Large Vision-Language Models by Interaction Guidance Sampling](https://mlsanity.com/paper/2507.05056/) (2025)
- [Bayesian Neural Networks Avoid Encoding Complex and Perturbation-Sensitive Concepts](https://mlsanity.com/paper/2302.13095/) (2023)
- [Where We Have Arrived in Proving the Emergence of Sparse Symbolic Concepts in AI Models](https://mlsanity.com/paper/2305.01939/) (2023)
- [Diagnosing the Compositional Knowledge of Vision Language Models from a Game-Theoretic View](https://mlsanity.com/paper/2405.17201/) (2024)
- [Towards Symbolic XAI -- Explanation Through Human Understandable Logical Relationships Between Features](https://mlsanity.com/paper/2408.17198/) (2024)
- [Explaining How a Neural Network Play the Go Game and Let People Learn](https://mlsanity.com/paper/2310.09838/) (2023)
- [Two-Phase Dynamics of Interactions Explains the Starting Point of a DNN Learning Over-Fitted Features](https://mlsanity.com/paper/2405.10262/) (2024)
- [Exploring Concept Depth: How Large Language Models Acquire Knowledge and Concept at Different Layers?](https://mlsanity.com/paper/2404.07066/) (2024)
- [Layerwise Change of Knowledge in Neural Networks](https://mlsanity.com/paper/2409.08712/) (2024)
- [Quantifying In-Context Reasoning Effects and Memorization Effects in LLMs](https://mlsanity.com/paper/2405.11880/) (2024)
- [NeurFlow: Interpreting Neural Networks through Neuron Groups and Functional Interactions](https://mlsanity.com/paper/2502.16105/) (2025)
- [Revisiting Generalization Power of a DNN in Terms of Symbolic Interactions](https://mlsanity.com/paper/2502.10162/) (2025)
- [Technical Report: Quantifying and Analyzing the Generalization Power of a DNN](https://mlsanity.com/paper/2505.06993/) (2025)
- [Attribution Explanations for Deep Neural Networks: A Theoretical Perspective](https://mlsanity.com/paper/2508.07636/) (2025)
- [Variational Language Concepts for Interpreting Foundation Language Models](https://mlsanity.com/paper/2410.03964/) (2024)
- [Why pre-training is beneficial for downstream classification tasks?](https://mlsanity.com/paper/2410.08455/) (2024)
- [Evaluating the Correctness of Inference Patterns Used by LLMs for Judgment](https://mlsanity.com/paper/2410.09083/) (2024)
- [The Interaction Bottleneck of Deep Neural Networks: Discovery, Proof, and Modulation](https://mlsanity.com/paper/2512.18607/) (2025)
- [Cross-LLM Consistency in Inference: Evidence from Shared Interactions](https://mlsanity.com/paper/2606.08129/) (2026)
- [Dissecting RGB-D Learning for Improved Multi-modal Fusion](https://mlsanity.com/paper/2308.10019/) (2023)

## Abstract (arXiv)

This paper aims to illustrate the concept-emerging phenomenon in a trained DNN. Specifically, we find that the inference score of a DNN can be disentangled into the effects of a few interactive concepts. These concepts can be understood as causal patterns in a sparse, symbolic causal graph, which explains the DNN. The faithfulness of using such a causal graph to explain the DNN is theoretically guaranteed, because we prove that the causal graph can well mimic the DNN's outputs on an exponential number of different masked samples. Besides, such a causal graph can be further simplified and re-written as an And-Or graph (AOG), without losing much explanation accuracy.