mlsanityData to 13 Oct 2026

Does a Neural Network Really Encode Symbolic Concepts?

arXiv 2302.13080 ↗First posted 25 Feb 2023cs.LG cs.AI cs.CVcited by 27

Not on the map. Why?

Abstract (arXiv)

Recently, a series of studies have tried to extract interactions between input variables modeled by a DNN and define such interactions as concepts encoded by the DNN. However, strictly speaking, there still lacks a solid guarantee whether such interactions indeed represent meaningful concepts. Therefore, in this paper, we examine the trustworthiness of interaction concepts from four perspectives. Extensive empirical studies have verified that a well-trained DNN usually encodes sparse, transferable, and discriminative concepts, which is partially aligned with human intuition.

Key facts. Cited by 27 papers in the map (data through 13 Oct 2026). Builds on 41 of its 78 references that are in the map. First posted on arXiv 25 Feb 2023.

Builds on

41 of 78 references.

References · 41

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,321
  3. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,713
  4. 2017A Unified Approach to Interpreting Model Predictionscited by 6,515
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,379
  6. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,231
  7. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,055
  8. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,389
  9. 2014Deep Learning Face Attributes in the Wildcited by 3,686
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,500
  11. 2017Axiomatic Attribution for Deep Networkscited by 3,368
  12. 2015Learning Deep Features for Discriminative Localizationcited by 2,434
29 more
  1. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,360
  2. 2013Visualizing and Understanding Convolutional Networkscited by 2,282
  3. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,316
  4. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,126
  5. 2018Neural Network Acceptability Judgmentscited by 957
  6. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 707
  7. 2017Interpretable Explanations of Black Boxes by Meaningful Perturbationcited by 611
  8. 2015Understanding Neural Networks Through Deep Visualizationcited by 333
  9. 2017Real Time Image Saliency for Black Box Classifierscited by 246
  10. 2017Distilling a Neural Network Into a Soft Decision Treecited by 235
  11. 2014Object Detectors Emerge in Deep Scene CNNscited by 178
  12. 2015Inverting Visual Representations with Convolutional Networkscited by 168
  13. 2019The Shapley Taylor Interaction Indexcited by 143
  14. 2019Explaining Deep Neural Networks with a Polynomial Time Algorithm for Shapley Values Approximationcited by 115
  15. 2020Explaining Explanations: Axiomatic Feature Interactions for Deep Networkscited by 103
  16. 2020A Unified Approach to Interpreting and Boosting Adversarial Transferabilitycited by 71
  17. 2017Beyond Sparsity: Tree Regularization of Deep Models for Interpretabilitycited by 68
  18. 2022Faith-Shap: The Faithful Shapley Interaction Indexcited by 68
  19. 2018Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMscited by 67
  20. 2018Hierarchical interpretations for neural network predictionscited by 66
  21. 2017Interpreting CNN Knowledge via an Explanatory Graphcited by 49
  22. 2021Discovering and Explaining the Representation Bottleneck of DNNscited by 42
  23. 2020Interpreting and Boosting Dropout from a Game-Theoretic Viewcited by 34
  24. 2018Explainable Neural Networks based on Additive Index Modelscited by 33
  25. 2021A Game-Theoretic Taxonomy of Visual Concepts in DNNscited by 16
  26. 2021Interpreting Attributions and Interactions of Adversarial Attackscited by 13
  27. 2021Interpreting Representation Quality of DNNs for 3D Point Cloud Processingcited by 12
  28. 2022Proving Common Mechanisms Shared by Twelve Methods of Boosting Adversarial Transferabilitycited by 12
  29. 2021A Hypothesis for the Aesthetic Appreciation in Neural Networkscited by 9

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 5; 2024: 14; 2025: 4; 2026: 4.

27 papers.

Sort:

By area:

  1. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 158
  2. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  3. 2024Explainable Generative AI (GenXAI): A Survey, Conceptualization, and Research Agendacited by 22
  4. 2024A Neuro-Symbolic Benchmark Suite for Concept Quality and Reasoning Shortcutscited by 19
  5. 2023HarsanyiNet: Computing Accurate Shapley Values in a Single Forward Propagationcited by 17
  6. 2024Defining and Extracting generalizable interaction primitives from DNNscited by 14
  7. 2024Towards the Dynamics of a DNN Learning Symbolic Interactionscited by 13
  8. 2023Can the Inference Logic of Large Language Models be Disentangled into Symbolic Concepts?cited by 11
  9. 2023Technical Note: Defining and Quantifying AND-OR Interactions for Faithful and Concise Explanation of DNNscited by 11
  10. 2023Where We Have Arrived in Proving the Emergence of Sparse Symbolic Concepts in AI Modelscited by 8
  11. 2024Diagnosing the Compositional Knowledge of Vision Language Models from a Game-Theoretic Viewcited by 7
  12. 2023Explaining How a Neural Network Play the Go Game and Let People Learncited by 6
15 more
  1. 2024Improving Semantic Control in Discrete Latent Spaces with Transformer Quantized Variational Autoencoderscited by 6
  2. 2024Two-Phase Dynamics of Interactions Explains the Starting Point of a DNN Learning Over-Fitted Featurescited by 5
  3. 2024Layerwise Change of Knowledge in Neural Networkscited by 4
  4. 2024Quantifying In-Context Reasoning Effects and Memorization Effects in LLMscited by 3
  5. 2025Revisiting Generalization Power of a DNN in Terms of Symbolic Interactionscited by 2
  6. 2025Technical Report: Quantifying and Analyzing the Generalization Power of a DNNcited by 2
  7. 2025Knowledge Transfer from Interaction Learningcited by 2
  8. 2024Why pre-training is beneficial for downstream classification tasks?cited by 1
  9. 2024Evaluating the Correctness of Inference Patterns Used by LLMs for Judgmentcited by 1
  10. 2026Cross-LLM Consistency in Inference: Evidence from Shared Interactionscited by 1
  11. 2024Disentangling Regional Primitives for Image Generationcited by 0
  12. 2025TaylorPODA: A Taylor Expansion-Based Method to Improve Post-Hoc Attributions for Opaque Modelscited by 0
  13. 2026Reconciling Contradictory Views on the Effectiveness of SFT in LLMs: An Interaction Perspectivecited by 0
  14. 2026A Unified Approach to Interpreting Knowledge Distillation for Large Language Models via Interactionscited by 0
  15. 2026Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactionscited by 0