mlsanityData to 7 Oct 2026

Revealing the Parametric Knowledge of Language Models: A Unified Framework for Attribution Methods

arXiv 2404.18655 ↗First posted 29 Apr 2024cs.CL cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

Language Models (LMs) acquire parametric knowledge from their training process, embedding it within their weights. The increasing scalability of LMs, however, poses significant challenges for understanding a model's inner workings and further for updating or correcting this embedded knowledge without the significant cost of retraining. This underscores the importance of unveiling exactly what knowledge is stored and its association with specific model components. Instance Attribution (IA) and Neuron Attribution (NA) offer insights into this training-acquired knowledge, though they have not been compared systematically. Our study introduces a novel evaluation framework to quantify and compare the knowledge revealed by IA and NA. To align the results of the methods we introduce the attribution method NA-Instances to apply NA for retrieving influential training instances, and IA-Neurons to discover important neurons of influential instances discovered by IA. We further propose a comprehensive list of faithfulness tests to evaluate the comprehensiveness and sufficiency of the explanations provided by both methods. Through extensive experiments and analysis, we demonstrate that NA generally reveals more diverse and comprehensive information regarding the LM's parametric knowledge compared to IA. Nevertheless, IA provides unique and valuable insights into the LM's parametric knowledge, which are not revealed by NA. Our findings further suggest the potential of a synergistic approach of combining the diverse findings of IA and NA for a more holistic understanding of an LM's parametric knowledge.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 21 of its 27 references that are in the map. First posted on arXiv 29 Apr 2024.

Builds on

21 of 27 references.

References · 21

  1. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  2. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  3. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  5. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  6. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  7. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  8. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  9. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  10. 2019Attention is not not Explanationcited by 629
  11. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  12. 2017Right for the Right Reasons: Training Differentiable Models by Constraining their Explanationscited by 293
9 more
  1. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 237
  2. 2020Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functionscited by 144
  3. 2023AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Webcited by 144
  4. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  5. 2021Neuron-level Interpretation of Deep NLP Models: A Surveycited by 81
  6. 2021Input Similarity from the Neural Network Perspectivecited by 78
  7. 2021Combining Feature and Instance Attribution to Detect Artifactscited by 38
  8. 2021An Empirical Comparison of Instance Attribution Methods for NLPcited by 35
  9. 2023Make Every Example Count: On the Stability and Utility of Self-Influence for Learning from Noisy NLP Datasetscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 2.

9 papers.

  1. 2024DYNAMICQA: Tracing Internal Knowledge Conflicts in Language Modelscited by 34
  2. 2024Evaluating Input Feature Explanations through a Unified Diagnostic Evaluation Frameworkcited by 5
  3. 2025Transparentize the Internal and External Knowledge Utilization in LLMs with Trustworthy Citationcited by 4
  4. 2026Taxonomy of the Retrieval System Framework: Pitfalls and Paradigmscited by 4
  5. 2025Guiding LLMs to Generate High-Fidelity and High-Quality Counterfactual Explanations for Text Classificationcited by 3
  6. 2025Knowledge-Level Consistency Reinforcement Learning: Dual-Fact Alignment for Long-Form Factualitycited by 3
  7. 2025Multi-Step Knowledge Interaction Analysis via Rank-2 Subspace Disentanglementcited by 1
  8. 2026Understanding the Interplay between LLMs' Utilisation of Parametric and Contextual Knowledge: A keynote at ECIR 2025cited by 1
  9. 2025Decoding Knowledge in Large Language Models: A Framework for Categorization and Comprehensioncited by 0