mlsanityData to 7 Oct 2026

Defining and Quantifying the Emergence of Sparse Concepts in DNNs

arXiv 2111.06206 ↗First posted 11 Nov 2021cs.LG cs.AI cs.CVcited by 36

Not on the map. Why?

Abstract (arXiv)

This paper aims to illustrate the concept-emerging phenomenon in a trained DNN. Specifically, we find that the inference score of a DNN can be disentangled into the effects of a few interactive concepts. These concepts can be understood as causal patterns in a sparse, symbolic causal graph, which explains the DNN. The faithfulness of using such a causal graph to explain the DNN is theoretically guaranteed, because we prove that the causal graph can well mimic the DNN's outputs on an exponential number of different masked samples. Besides, such a causal graph can be further simplified and re-written as an And-Or graph (AOG), without losing much explanation accuracy.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 54 of its 95 references that are in the map. First posted on arXiv 11 Nov 2021.

Builds on

54 of 95 references.

References · 54

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  4. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  6. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  7. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  8. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  9. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  10. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  11. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  12. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
38 more
  1. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,124
  2. 2018Neural Network Acceptability Judgmentscited by 952
  3. 2017Interpretable Explanations of Black Boxes by Meaningful Perturbationcited by 611
  4. 2020Captum: A unified and generic model interpretability library for PyTorchcited by 542
  5. 2015Understanding Neural Networks Through Deep Visualizationcited by 332
  6. 2016Not Just a Black Box: Learning Important Features Through Propagating Activation Differencescited by 295
  7. 2018Consistent Individualized Feature Attribution for Tree Ensemblescited by 279
  8. 2019Understanding Deep Networks via Extremal Perturbations and Smooth Maskscited by 257
  9. 2018Learning to Explain: An Information-Theoretic Perspective on Model Interpretationcited by 248
  10. 2017Real Time Image Saliency for Black Box Classifierscited by 246
  11. 2017Distilling a Neural Network Into a Soft Decision Treecited by 230
  12. 2014Object Detectors Emerge in Deep Scene CNNscited by 178
  13. 2015Inverting Visual Representations with Convolutional Networkscited by 168
  14. 2018Abduction-Based Explanations for Machine Learning Modelscited by 159
  15. 2020Improving KernelSHAP: Practical Shapley Value Estimation via Linear Regressioncited by 153
  16. 2019The Shapley Taylor Interaction Indexcited by 143
  17. 2019Explaining Deep Neural Networks with a Polynomial Time Algorithm for Shapley Values Approximationcited by 115
  18. 2019Asymmetric Shapley values: incorporating causal knowledge into model-agnostic explainabilitycited by 114
  19. 2020Causal Shapley Values: Exploiting Causal Knowledge to Explain Individual Predictions of Complex Modelscited by 107
  20. 2020Explaining Explanations: Axiomatic Feature Interactions for Deep Networkscited by 103
  21. 2021The Causal-Neural Connection: Expressiveness, Learnability, and Inferencecited by 97
  22. 2019Neural Network Attributions: A Causal Perspectivecited by 77
  23. 2020A Unified Approach to Interpreting and Boosting Adversarial Transferabilitycited by 71
  24. 2017Beyond Sparsity: Tree Regularization of Deep Models for Interpretabilitycited by 68
  25. 2018Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMscited by 67
  26. 2018Hierarchical interpretations for neural network predictionscited by 66
  27. 2017A causal framework for explaining the predictions of black-box sequence-to-sequence modelscited by 58
  28. 2020Shapley Flow: A Graph-based Approach to Interpreting Model Predictionscited by 51
  29. 2021Entropy-based Logic Explanations of Neural Networkscited by 51
  30. 2017Interpreting CNN Knowledge via an Explanatory Graphcited by 49
  31. 2021Scalable Rule-Based Representation Learning for Interpretable Classificationcited by 40
  32. 2021Explanations for Monotonic Classifierscited by 39
  33. 2020Interpreting and Boosting Dropout from a Game-Theoretic Viewcited by 34
  34. 2018Explainable Neural Networks based on Additive Index Modelscited by 33
  35. 2018Causal Learning and Explanation of Deep Neural Networks via Autoencoded Activationscited by 23
  36. 2021A Game-Theoretic Taxonomy of Visual Concepts in DNNscited by 16
  37. 2021Interpreting Attributions and Interactions of Adversarial Attackscited by 13
  38. 2021Can We Faithfully Represent Masked States to Compute Shapley Values on a DNN?cited by 10

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 16; 2025: 6; 2026: 6.

36 papers.

Sort:

By area:

  1. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  2. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  3. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  4. 2024Identifying Semantic Induction Heads to Understand In-Context Learningcited by 46
  5. 2023Explaining Generalization Power of a DNN Using Interactive Conceptscited by 25
  6. 2023HarsanyiNet: Computing Accurate Shapley Values in a Single Forward Propagationcited by 16
  7. 2024Defining and Extracting generalizable interaction primitives from DNNscited by 14
  8. 2024Towards the Dynamics of a DNN Learning Symbolic Interactionscited by 13
  9. 2023Can the Inference Logic of Large Language Models be Disentangled into Symbolic Concepts?cited by 11
  10. 2023Technical Note: Defining and Quantifying AND-OR Interactions for Faithful and Concise Explanation of DNNscited by 10
  11. 2025INTER: Mitigating Hallucination in Large Vision-Language Models by Interaction Guidance Samplingcited by 10
  12. 2023Bayesian Neural Networks Avoid Encoding Complex and Perturbation-Sensitive Conceptscited by 9
24 more
  1. 2023Where We Have Arrived in Proving the Emergence of Sparse Symbolic Concepts in AI Modelscited by 8
  2. 2024Diagnosing the Compositional Knowledge of Vision Language Models from a Game-Theoretic Viewcited by 7
  3. 2024Towards Symbolic XAI -- Explanation Through Human Understandable Logical Relationships Between Featurescited by 7
  4. 2023Explaining How a Neural Network Play the Go Game and Let People Learncited by 6
  5. 2024Two-Phase Dynamics of Interactions Explains the Starting Point of a DNN Learning Over-Fitted Featurescited by 5
  6. 2024Exploring Concept Depth: How Large Language Models Acquire Knowledge and Concept at Different Layers?cited by 4
  7. 2024Layerwise Change of Knowledge in Neural Networkscited by 4
  8. 2024Quantifying In-Context Reasoning Effects and Memorization Effects in LLMscited by 3
  9. 2025NeurFlow: Interpreting Neural Networks through Neuron Groups and Functional Interactionscited by 3
  10. 2025Attribution Explanations for Deep Neural Networks: A Theoretical Perspectivecited by 2
  11. 2024Variational Language Concepts for Interpreting Foundation Language Modelscited by 1
  12. 2024Why pre-training is beneficial for downstream classification tasks?cited by 1
  13. 2024Evaluating the Correctness of Inference Patterns Used by LLMs for Judgmentcited by 1
  14. 2025Revisiting Generalization Power of a DNN in Terms of Symbolic Interactionscited by 1
  15. 2025Technical Report: Quantifying and Analyzing the Generalization Power of a DNNcited by 1
  16. 2025The Interaction Bottleneck of Deep Neural Networks: Discovery, Proof, and Modulationcited by 1
  17. 2026Cross-LLM Consistency in Inference: Evidence from Shared Interactionscited by 1
  18. 2023Dissecting RGB-D Learning for Improved Multi-modal Fusioncited by 0
  19. 2024Disentangling Regional Primitives for Image Generationcited by 0
  20. 2026Hierarchical Sparse Circuit Extraction from Billion-Parameter Language Models through Scalable Attribution Graph Decompositioncited by 0
  21. 2026Reconciling Contradictory Views on the Effectiveness of SFT in LLMs: An Interaction Perspectivecited by 0
  22. 2026A Unified Approach to Interpreting Knowledge Distillation for Large Language Models via Interactionscited by 0
  23. 2026Saturation Makes Quantization Error Additive: A Coverage Model with a Certificatecited by 0
  24. 2026Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactionscited by 0