mlsanityData to 14 Oct 2026

The Attribution Contract for Generative Language Models

arXiv 2605.23080 ↗First posted 21 May 2026cs.LGcited by 0

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Feature attribution scores each part of an input by how much it explains a model's output. We argue that in generative language models these scores carry no fixed meaning. A classifier has a single output to explain, but a generative model produces its output token by token, and each generated token is both an output and an input, so explaining the output becomes several distinct questions. We support this claim by proposing the Attribution Contract, a framework that names the question a set of attribution scores answers. A contract specifies the model score being explained, what is held fixed, the target output, the generation process, and which features can receive attribution, choices that matter in practice. For example, eligible features decide what a score explains: under a local next-token contract, the model's own generated tokens receive 29% of the attribution mass, comparable to the 25% on the input question, and reading this as a prompt-level explanation is a mistake we name the self-attribution fallacy. The generative process also shapes feature attribution: on a mixture-of-experts model, Integrated Gradients completeness is unreliable, since recomputing expert routing along the attribution path leaves a large residual that holding the routing fixed substantially lowers, while a masked-diffusion model, which fixes its choices at generation, has a residual that instead decreases with more steps. These results show that Attribution Contracts should be carefully specified when proposing and evaluating attribution methods for generative language models, since the meaning and evaluation of an attribution depend on its explanatory setting.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 33 of its 36 references that are in the map. First posted on arXiv 21 May 2026.

Builds on

33 of 36 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024OLMoE: Open Mixture-of-Experts Language Models3 Sep
  2. 2024Mixtral of Experts8 Jan

Other references · 31

  1. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  2. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  4. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  5. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  6. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  7. 2022Red Teaming Language Models with Language Modelscited by 1,026
  8. 2025Large Language Diffusion Modelscited by 912
  9. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 745
  10. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  11. 2016The Mythos of Model Interpretabilitycited by 721
  12. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 599
19 more
  1. 2015Evaluating the visualization of what a Deep Neural Network has learnedcited by 419
  2. 2020True to the Model or True to the Data?cited by 126
  3. 2024ContextCite: Attributing Model Generation to Contextcited by 98
  4. 2022Interpreting Language Models with Contrastive Explanationscited by 92
  5. 2021The effectiveness of feature attribution methods and its correlation with automatic evaluation scorescited by 88
  6. 2022Impossibility Theorems for Feature Attributioncited by 78
  7. 2023Inseq: An Interpretability Toolkit for Sequence Generation Modelscited by 78
  8. 2022Post hoc Explanations may be Ineffective for Detecting Unknown Spurious Correlationcited by 59
  9. 2023Sequential Integrated Gradients: a simple but effective method for explaining language modelscited by 52
  10. 2021Discretized Integrated Gradients for Explaining Language Modelscited by 46
  11. 2023Using Captum to Explain Generative Language Modelscited by 33
  12. 2025Continuous Diffusion Model for Language Modelingcited by 31
  13. 2024Interpretable LLM-based Table Question Answeringcited by 18
  14. 2024ReAGent: A Model-agnostic Feature Attribution Method for Generative Language Modelscited by 12
  15. 2025The Dead Salmons of AI Interpretabilitycited by 12
  16. 2023Faithfulness Measurable Masked Language Modelscited by 5
  17. 2026Jacobian Scopes: token-level causal attributions in LLMscited by 4
  18. 2025Explaining the Reasoning of Large Language Models Using Attribution Graphscited by 3
  19. 2026Hessian-Enhanced Token Attribution (HETA): Interpreting Autoregressive LLMscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Probing for Knowledge Attribution in Large Language Models
  2. 2026Matryoshka attribution: Learning to attribute language model outputs to representations and weights
  3. 2025Explanation Bias is a Product: Revealing the Hidden Lexical and Position Preferences in Post-Hoc Feature Attribution
  4. 2026The Metagame of Interpretability and Meta-Attributions
  5. 2024Advancing Large Language Model Attribution through Self-Improving
  6. 2026When Attribution Patching Lies: Diagnosis and a Second-Order Correction
  7. 2026Concept Influence: Leveraging Interpretability to Improve Performance and Efficiency in Training Data Attribution
  8. 2026The Attribution Blind Spot: Detecting When Language Models Rely on Memory Rather Than Retrieved Context