mlsanityData to 7 Oct 2026

Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking

arXiv 2402.14811 ↗First posted 22 Feb 2024cs.CL cs.LGcited by 140

Not on the map. Why?

Abstract (arXiv)

Fine-tuning on generalized tasks such as instruction following, code generation, and mathematics has been shown to enhance language models' performance on a range of tasks. Nevertheless, explanations of how such fine-tuning influences the internal computations in these models remain elusive. We study how fine-tuning affects the internal mechanisms implemented in language models. As a case study, we explore the property of entity tracking, a crucial facet of language comprehension, where models fine-tuned on mathematics have substantial performance gains. We identify the mechanism that enables entity tracking and show that (i) in both the original model and its fine-tuned versions primarily the same circuit implements entity tracking. In fact, the entity tracking circuit of the original model on the fine-tuned versions performs better than the full original model. (ii) The circuits of all the models implement roughly the same functionality: Entity tracking is performed by tracking the position of the correct entity in both the original model and its fine-tuned versions. (iii) Performance boost in the fine-tuned models is primarily attributed to its improved ability to handle the augmented positional information. To uncover these findings, we employ: Patch Patching, DCM, which automatically detects model components responsible for specific semantics, and CMAP, a new approach for patching activations across models to reveal improved mechanisms. Our findings suggest that fine-tuning enhances, rather than fundamentally alters, the mechanistic operation of the model.

Key facts. Cited by 140 papers in the map (data through 7 Oct 2026). Builds on 35 of its 51 references that are in the map. First posted on arXiv 22 Feb 2024.

Builds on

35 of 51 references.

References · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  9. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  10. 2023LIMA: Less Is More for Alignmentcited by 1,080
  11. 2022Editing Models with Task Arithmeticcited by 1,076
  12. 2022Toy Models of Superpositioncited by 928
23 more
  1. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  2. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  3. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  4. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  5. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
  6. 2019Revealing the Dark Secrets of BERTcited by 300
  7. 2023Localizing Model Behavior with Path Patchingcited by 182
  8. 2021Implicit Representations of Meaning in Neural Language Modelscited by 179
  9. 2022Language Models of Code are Few-Shot Commonsense Learnerscited by 160
  10. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 145
  11. 2023A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operationscited by 141
  12. 2020What Happens To BERT Embeddings During Fine-tuning?cited by 113
  13. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
  14. 2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 104
  15. 2023Interpretability at Scale: Identifying Causal Mechanisms in Alpacacited by 102
  16. 2023Tracr: Compiled Transformers as a Laboratory for Interpretabilitycited by 99
  17. 2021Chess as a Testbed for Language Model State Trackingcited by 69
  18. 2022GPT Takes the Bar Examcited by 65
  19. 2023Knowledge is a Region in Weight Space for Fine-tuned Language Modelscited by 58
  20. 2020Similarity Analysis of Contextual Word Representation Modelscited by 54
  21. 2023Entity Tracking in Language Modelscited by 44
  22. 2021A Closer Look at How Fine-tuning Changes BERTcited by 42
  23. 2023Discovering Variable Binding Circuitry with Desideratacited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 36; 2025: 50; 2026: 54.

140 papers.

Sort:

By area:

  1. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  2. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  3. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 412
  4. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  5. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 153
  6. 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 149
  7. 2024Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanismscited by 144
  8. 2024Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Modelscited by 114
  9. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  10. 2024Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristicscited by 94
  11. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  12. 2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 82
38 more
  1. 2025Persona Features Control Emergent Misalignmentcited by 76
  2. 2025LLMs Encode Harmfulness and Refusal Separatelycited by 75
  3. 2024LLM Circuit Analyses Are Consistent Across Training and Scalecited by 60
  4. 2024Finding Transformer Circuits with Edge Pruningcited by 58
  5. 2024What Makes and Breaks Safety Fine-tuning? A Mechanistic Studycited by 54
  6. 2025MIB: A Mechanistic Interpretability Benchmarkcited by 49
  7. 2024Who's asking? User personas and the mechanics of latent misalignmentcited by 40
  8. 2024Monitoring Latent World States in Language Models with Propositional Probescited by 39
  9. 2025Language Models use Lookbacks to Track Beliefscited by 36
  10. 2024Answer, Assemble, Ace: Understanding How LMs Answer Multiple Choice Questionscited by 35
  11. 2025Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuningcited by 34
  12. 2025PaTH Attention: Position Encoding via Accumulating Householder Transformationscited by 32
  13. 2024Code Pretraining Improves Entity Tracking Abilities of Language Modelscited by 31
  14. 2025(How) Do Language Models Track State?cited by 31
  15. 2025Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuningcited by 31
  16. 2024No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attackscited by 28
  17. 2024Representational Analysis of Binding in Language Modelscited by 27
  18. 2024The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysiscited by 24
  19. 2024U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills in LLMscited by 24
  20. 2025Mixing Mechanisms: How Language Models Retrieve Bound Entities In-Contextcited by 24
  21. 2025Narrow Finetuning Leaves Clearly Readable Traces in Activation Differencescited by 24
  22. 2024Relational Composition in Neural Networks: A Survey and Call to Actioncited by 23
  23. 2024Noise Injection Reveals Hidden Capabilities of Sandbagging Language Modelscited by 22
  24. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 22
  25. 2025EAP-GP: Mitigating Saturation Effect in Gradient-based Automated Circuit Identificationcited by 21
  26. 2025Towards Understanding Fine-Tuning Mechanisms of LLMs via Circuit Analysiscited by 20
  27. 2025How Do Transformers Learn Variable Binding in Symbolic Programs?cited by 19
  28. 2024Bayesian scaling laws for in-context learningcited by 18
  29. 2025Position-aware Automatic Circuit Discoverycited by 14
  30. 2025Implicit Reasoning in Transformers is Reasoning through Shortcutscited by 14
  31. 2025Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Themcited by 14
  32. 2025Mechanistic Interpretability as Statistical Estimation: A Variance Analysiscited by 14
  33. 2025The Geometry of Self-Verification in a Task-Specific Reasoning Modelcited by 12
  34. 2024Understanding Multimodal LLMs: the Mechanistic Interpretability of Llava in Visual Question Answeringcited by 11
  35. 2025Are formal and functional linguistic mechanisms dissociated in language models?cited by 11
  36. 2024Understanding Synthetic Context Extension via Retrieval Headscited by 10
  37. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
  38. 2025RelP: Faithful and Efficient Circuit Discovery in Language Models via Relevance Patchingcited by 10

50 most cited of 140 All 140 (JSON)