mlsanityData to 14 Oct 2026

Neuroplasticity and Corruption in Model Mechanisms: A Case Study Of Indirect Object Identification

arXiv 2503.01896 ↗First posted 27 Feb 2025cs.LG cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Previous research has shown that fine-tuning language models on general tasks enhance their underlying mechanisms. However, the impact of fine-tuning on poisoned data and the resulting changes in these mechanisms are poorly understood. This study investigates the changes in a model's mechanisms during toxic fine-tuning and identifies the primary corruption mechanisms. We also analyze the changes after retraining a corrupted model on the original dataset and observe neuroplasticity behaviors, where the model relearns original mechanisms after fine-tuning the corrupted model. Our findings indicate that: (i) Underlying mechanisms are amplified across task-specific fine-tuning which can be generalized to longer epochs, (ii) Model corruption via toxic fine-tuning is localized to specific circuit components, (iii) Models exhibit neuroplasticity when retraining corrupted models on clean dataset, reforming the original model mechanisms.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 50 of its 65 references that are in the map. First posted on arXiv 27 Feb 2025.

Builds on

50 of 65 references.

References · 50

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2017Deep reinforcement learning from human preferencescited by 3,912
  4. 2016Pointer Sentinel Mixture Modelscited by 3,096
  5. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  6. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  7. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  8. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  9. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  10. 2022Toy Models of Superpositioncited by 971
  11. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  12. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
38 more
  1. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  2. 2021Causal Abstractions of Neural Networkscited by 422
  3. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 359
  4. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 290
  5. 2023Poisoning Language Models During Instruction Tuningcited by 269
  6. 2023Attribution Patching Outperforms Automated Circuit Discoverycited by 209
  7. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 202
  8. 2023Localizing Model Behavior with Path Patchingcited by 195
  9. 2024Improving Dictionary Learning with Gated Sparse Autoencoderscited by 179
  10. 2020Witches' Brew: Industrial Scale Data Poisoning via Gradient Matchingcited by 163
  11. 2022Language Models of Code are Few-Shot Commonsense Learnerscited by 160
  12. 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 156
  13. 2024Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Trackingcited by 149
  14. 2024Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanismscited by 149
  15. 2023A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operationscited by 144
  16. 2020MetaPoison: Practical General-purpose Clean-label Data Poisoningcited by 120
  17. 2023On the Exploitability of Instruction Tuningcited by 116
  18. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 115
  19. 2023Circuit Component Reuse Across Tasks in Transformer Language Modelscited by 108
  20. 2023Tracr: Compiled Transformers as a Laboratory for Interpretabilitycited by 104
  21. 2024BadEdit: Backdooring large language models by model editingcited by 96
  22. 2023Copy Suppression: Comprehensively Understanding an Attention Headcited by 76
  23. 2022Neuro-Symbolic Language Modeling with Automaton-augmented Retrievalcited by 60
  24. 2023Knowledge is a Region in Weight Space for Fine-tuned Language Modelscited by 58
  25. 2024Explorations of Self-Repair in Language Modelscited by 39
  26. 2024A Survey on Generative AI and LLM for Video Generation, Understanding, and Streamingcited by 37
  27. 2024Unveiling the Generalization Power of Fine-Tuned Large Language Modelscited by 36
  28. 2023Discovering Variable Binding Circuitry with Desideratacited by 34
  29. 2021Defending Against Backdoor Attacks in Natural Language Generationcited by 33
  30. 2023Is Fine-tuning Needed? Pre-trained Language Models Are Near Perfect for Out-of-Domain Detectioncited by 29
  31. 2024Large Language Models Relearn Removed Conceptscited by 27
  32. 2024Large Language Models and Causal Inference in Collaboration: A Surveycited by 26
  33. 2021What Doesn't Kill You Makes You Robust(er): How to Adversarially Train against Data Poisoningcited by 22
  34. 2023ParaFuzz: An Interpretability-Driven Technique for Detecting Poisoned Samples in NLPcited by 19
  35. 2023Setting the Trap: Capturing and Defeating Backdoors in Pretrained Language Models through Honeypotscited by 19
  36. 2024Defending Against Weight-Poisoning Backdoor Attacks for Parameter-Efficient Fine-Tuningcited by 16
  37. 2024How does GPT-2 Predict Acronyms? Extracting and Understanding a Circuit via Mechanistic Interpretabilitycited by 14
  38. 2024Your Finetuned Large Language Model is Already a Powerful Out-of-distribution Detectorcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Towards Understanding and Improving Refusal in Compressed Models via Mechanistic Interpretabilitycited by 7
  2. 2025Planted in Pretraining, Swayed by Finetuning: A Case Study on the Origins of Cognitive Biases in LLMscited by 5
  3. 2026Beyond Static Interpretability: Anticipating Post-SFT Mechanisms from Pre-SFT Parameters for Better Tuningcited by 0