mlsanityData to 7 Oct 2026

Can the Inference Logic of Large Language Models be Disentangled into Symbolic Concepts?

arXiv 2304.01083 ↗First posted 3 Apr 2023cs.CL cs.AI cs.CVcited by 11

Not on the map. Why?

Abstract (arXiv)

In this paper, we explain the inference logic of large language models (LLMs) as a set of symbolic concepts. Many recent studies have discovered that traditional DNNs usually encode sparse symbolic concepts. However, because an LLM has much more parameters than traditional DNNs, whether the LLM also encodes sparse symbolic concepts is still an open problem. Therefore, in this paper, we propose to disentangle the inference score of LLMs for dialogue tasks into a small number of symbolic concepts. We verify that we can use those sparse concepts to well estimate all inference scores of the LLM on all arbitrarily masking states of the input sentence. We also evaluate the transferability of concepts encoded by an LLM and verify that symbolic concepts usually exhibit high transferability across similar input sentences. More crucially, those symbolic concepts can be used to explain the exact reasons accountable for the LLM's prediction errors.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 15 of its 19 references that are in the map. First posted on arXiv 3 Apr 2023.

Builds on

15 of 19 references.

References · 15

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  3. 2019The Shapley Taylor Interaction Indexcited by 143
  4. 2020A Unified Approach to Interpreting and Boosting Adversarial Transferabilitycited by 71
  5. 2021Discovering and Explaining the Representation Bottleneck of DNNscited by 42
  6. 2021Defining and Quantifying the Emergence of Sparse Concepts in DNNscited by 36
  7. 2020Interpreting and Boosting Dropout from a Game-Theoretic Viewcited by 34
  8. 2023Does a Neural Network Really Encode Symbolic Concepts?cited by 27
  9. 2023Explaining Generalization Power of a DNN Using Interactive Conceptscited by 25
  10. 2021A Game-Theoretic Taxonomy of Visual Concepts in DNNscited by 16
  11. 2021Interpreting Attributions and Interactions of Adversarial Attackscited by 13
  12. 2022Proving Common Mechanisms Shared by Twelve Methods of Boosting Adversarial Transferabilitycited by 12
3 more
  1. 2023Understanding and Unifying Fourteen Attribution Methods with Taylor Interactionscited by 11
  2. 2021Can We Faithfully Represent Masked States to Compute Shapley Values on a DNN?cited by 10
  3. 2021A Hypothesis for the Aesthetic Appreciation in Neural Networkscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 5; 2025: 3; 2026: 1.

11 papers.

  1. 2024Defining and Extracting generalizable interaction primitives from DNNscited by 14
  2. 2023Technical Note: Defining and Quantifying AND-OR Interactions for Faithful and Concise Explanation of DNNscited by 10
  3. 2023Where We Have Arrived in Proving the Emergence of Sparse Symbolic Concepts in AI Modelscited by 8
  4. 2024Layerwise Change of Knowledge in Neural Networkscited by 4
  5. 2024Quantifying In-Context Reasoning Effects and Memorization Effects in LLMscited by 3
  6. 2025Knowledge Transfer from Interaction Learningcited by 2
  7. 2024Evaluating the Correctness of Inference Patterns Used by LLMs for Judgmentcited by 1
  8. 2025Revisiting Generalization Power of a DNN in Terms of Symbolic Interactionscited by 1
  9. 2025Technical Report: Quantifying and Analyzing the Generalization Power of a DNNcited by 1
  10. 2024Disentangling Regional Primitives for Image Generationcited by 0
  11. 2026A Unified Approach to Interpreting Knowledge Distillation for Large Language Models via Interactionscited by 0