mlsanityData to 7 Oct 2026

Learning to Interpret Weight Differences in Language Models

arXiv 2510.05092 ↗First posted 6 Oct 2025cs.LG cs.AI cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

Finetuning (pretrained) language models is a standard approach for updating their internal parametric knowledge and specializing them to new tasks and domains. However, the corresponding model weight changes ("weight diffs") are not generally interpretable. While inspecting the finetuning dataset can give a sense of how the model might have changed, these datasets are often not publicly available or are too large to work with directly. Towards the goal of comprehensively understanding weight diffs in natural language, we introduce Diff Interpretation Tuning (DIT), a method that trains models to describe their own finetuning-induced modifications. Our approach uses synthetic, labeled weight diffs to train a DIT-adapter, which can be applied to a compatible finetuned model to make it describe how it has changed. We demonstrate in two proof-of-concept settings (reporting hidden behaviors and summarizing finetuned knowledge) that our method enables models to describe their finetuning-induced modifications using accurate natural language descriptions.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 30 of its 53 references that are in the map. First posted on arXiv 6 Oct 2025.

Builds on

30 of 53 references.

References · 30

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  4. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  5. 2022Editing Models with Task Arithmeticcited by 1,076
  6. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
  7. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 293
  8. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 225
  9. 2024Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Trackingcited by 140
  10. 2020What Happens To BERT Embeddings During Fine-tuning?cited by 113
  11. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
  12. 2024Looking Inward: Language Models Can Learn About Themselves by Introspectioncited by 105
18 more
  1. 2025Tell me about yourself: LLMs are aware of their learned behaviorscited by 86
  2. 2024SelfIE: Self-Interpretation of Large Language Model Embeddingscited by 63
  3. 2023Knowledge is a Region in Weight Space for Fine-tuned Language Modelscited by 58
  4. 2022Describing Differences between Text Distributions with Natural Languagecited by 54
  5. 2024MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmeticcited by 53
  6. 2024LatentQA: Teaching LLMs to Decode Activations Into Natural Languagecited by 49
  7. 2020The TrojAI Software Framework: An OpenSource tool for Embedding Trojans into Deep Learning Modelscited by 41
  8. 2025Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activationscited by 37
  9. 2025Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuningcited by 31
  10. 2021Fine-Tuned Transformers Show Clusters of Similar Representations Across Layerscited by 29
  11. 2024Competition Report: Finding Universal Jailbreak Backdoors in Aligned LLMscited by 27
  12. 2025Eliciting Language Model Behaviors with Investigator Agentscited by 21
  13. 2023Fine-tuning Happens in Tiny Subspaces: Exploring Intrinsic Task-specific Subspaces of Pre-trained Language Modelscited by 19
  14. 2025Analyzing Finetuning Representation Shift for Multimodal LLMs Steeringcited by 15
  15. 2024Trojan Detection in Large Language Models: Insights from The Trojan Detection Challengecited by 14
  16. 2023On Robustness of Finetuned Transformer-based NLP Modelscited by 8
  17. 2025Do Activation Verbalization Methods Convey Privileged Information?cited by 8
  18. 2025ConfGuard: A Simple and Effective Backdoor Detection for Large Language Modelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 8.

9 papers.

  1. 2025Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainerscited by 41
  2. 2026Introspection Adapters: Training LLMs to Report Their Learned Behaviorscited by 8
  3. 2026QuantLRM: Quantization of Large Reasoning Models via Fine-Tuning Signalscited by 5
  4. 2026Introspective Coupling: Self-Explanation Training Tracks Behavioral Change Despite Fixed Supervisioncited by 3
  5. 2026Position: It's Time to Optimize LLMs for Self-Consistencycited by 1
  6. 2026Detecting CSAM Text-to-Image LoRAs From Weightscited by 0
  7. 2026Introspecting Alignment Shifts Beyond Behaviors Implanted Through Fine-Tuningcited by 0
  8. 2026In-Context Neurofeedback: Can LLMs Control Their Internal Representations through Privileged Access?cited by 0
  9. 2026Imprint Reader: From Weight-Update Readout to Behavioral Interventioncited by 0