mlsanityData to 7 Oct 2026

Where We Have Arrived in Proving the Emergence of Sparse Symbolic Concepts in AI Models

arXiv 2305.01939 ↗First posted 3 May 2023cs.LG cs.AI cs.CVcited by 8

Not on the map. Why?

Abstract (arXiv)

This study aims to prove the emergence of symbolic concepts (or more precisely, sparse primitive inference patterns) in well-trained deep neural networks (DNNs). Specifically, we prove the following three conditions for the emergence. (i) The high-order derivatives of the network output with respect to the input variables are all zero. (ii) The DNN can be used on occluded samples and when the input sample is less occluded, the DNN will yield higher confidence. (iii) The confidence of the DNN does not significantly degrade on occluded samples. These conditions are quite common, and we prove that under these conditions, the DNN will only encode a relatively small number of sparse interactions between input variables. Moreover, we can consider such interactions as symbolic primitive inference patterns encoded by a DNN, because we show that inference scores of the DNN on an exponentially large number of randomly masked samples can always be well mimicked by numerical effects of just a few interactions.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 28 of its 51 references that are in the map. First posted on arXiv 3 May 2023.

Builds on

28 of 51 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  5. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  10. 2018Neural Network Acceptability Judgmentscited by 952
  11. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 703
  12. 2021ResMLP: Feedforward networks for image classification with data-efficient trainingcited by 338
16 more
  1. 2020Understanding the Role of Individual Units in a Deep Neural Networkcited by 254
  2. 2019The Shapley Taylor Interaction Indexcited by 143
  3. 2020The Hessian Penalty: A Weak Prior for Unsupervised Disentanglementcited by 90
  4. 2020A Unified Approach to Interpreting and Boosting Adversarial Transferabilitycited by 71
  5. 2022Faith-Shap: The Faithful Shapley Interaction Indexcited by 68
  6. 2021Discovering and Explaining the Representation Bottleneck of DNNscited by 42
  7. 2021Defining and Quantifying the Emergence of Sparse Concepts in DNNscited by 36
  8. 2020Interpreting and Boosting Dropout from a Game-Theoretic Viewcited by 34
  9. 2023Does a Neural Network Really Encode Symbolic Concepts?cited by 27
  10. 2023Explaining Generalization Power of a DNN Using Interactive Conceptscited by 25
  11. 2021A Game-Theoretic Taxonomy of Visual Concepts in DNNscited by 16
  12. 2022Proving Common Mechanisms Shared by Twelve Methods of Boosting Adversarial Transferabilitycited by 12
  13. 2023Understanding and Unifying Fourteen Attribution Methods with Taylor Interactionscited by 11
  14. 2023Can the Inference Logic of Large Language Models be Disentangled into Symbolic Concepts?cited by 11
  15. 2023Technical Note: Defining and Quantifying AND-OR Interactions for Faithful and Concise Explanation of DNNscited by 10
  16. 2021A Hypothesis for the Aesthetic Appreciation in Neural Networkscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 3; 2025: 1; 2026: 2.

8 papers.

  1. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  2. 2024Defining and Extracting generalizable interaction primitives from DNNscited by 14
  3. 2023A Theoretical Explanation of Activation Sparsity through Flat Minima and Adversarial Robustnesscited by 7
  4. 2023Explaining How a Neural Network Play the Go Game and Let People Learncited by 6
  5. 2024Quantifying In-Context Reasoning Effects and Memorization Effects in LLMscited by 3
  6. 2025Technical Report: Quantifying and Analyzing the Generalization Power of a DNNcited by 1
  7. 2026Hierarchical Sparse Circuit Extraction from Billion-Parameter Language Models through Scalable Attribution Graph Decompositioncited by 0
  8. 2026Towards the Connection between Activation Sparsity and Flat Minimacited by 0