mlsanityData to 7 Oct 2026

Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks

arXiv 2311.12786 ↗First posted 21 Nov 2023cs.LGcited by 110

Not on the map. Why?

Cited by 28 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Fine-tuning large pre-trained models has become the de facto strategy for developing both task-specific and general-purpose machine learning systems, including developing models that are safe to deploy. Despite its clear importance, there has been minimal work that explains how fine-tuning alters the underlying capabilities learned by a model during pretraining: does fine-tuning yield entirely novel capabilities or does it just modulate existing ones? We address this question empirically in synthetic, controlled settings where we can use mechanistic interpretability tools (e.g., network pruning and probing) to understand how the model's underlying capabilities are changing. We perform an extensive analysis of the effects of fine-tuning in these settings, and show that: (i) fine-tuning rarely alters the underlying model capabilities; (ii) a minimal transformation, which we call a 'wrapper', is typically learned on top of the underlying model capabilities, creating the illusion that they have been modified; and (iii) further fine-tuning on a task where such hidden capabilities are relevant leads to sample-efficient 'revival' of the capability, i.e., the model begins reusing these capability after only a few gradient steps. This indicates that practitioners can unintentionally remove a model's safety wrapper merely by fine-tuning it on a, e.g., superficially unrelated, downstream task. We additionally perform analysis on language models trained on the TinyStories dataset to support our claims in a more realistic setup.

Key facts. Cited by 110 papers in the map (data through 7 Oct 2026). Builds on 76 of its 98 references that are in the map. First posted on arXiv 21 Nov 2023.

Builds on

76 of 98 references.

References · 76

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  11. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  12. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
38 more
  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  2. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  3. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  4. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  5. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  6. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  7. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  8. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  9. 2021Ethical and social risks of harm from Language Modelscited by 957
  10. 2019Attention is not Explanationcited by 916
  11. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  12. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  13. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  14. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  15. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  16. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  17. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  18. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  19. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  20. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  21. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  22. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  23. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
  24. 2019Is Attention Interpretable?cited by 387
  25. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 349
  26. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 313
  27. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  28. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 293
  29. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 284
  30. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 264
  31. 2022Transformers Learn Shortcuts to Automatacited by 260
  32. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 230
  33. 2022Neural Networks and the Chomsky Hierarchycited by 214
  34. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 198
  35. 2020On the Theory of Transfer Learning: The Importance of Task Diversitycited by 186
  36. 2020Feature Learning in Infinite-Width Neural Networkscited by 182
  37. 2021Thinking Like Transformerscited by 182
  38. 2023MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbotscited by 172

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 47; 2025: 34; 2026: 27.

110 papers.

Sort:

By area:

  1. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  2. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  3. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  4. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 220
  5. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  6. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 179
  7. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  8. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 153
  9. 2024Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Trackingcited by 140
  10. 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 131
  11. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  12. 2024Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Modelscited by 114
38 more
  1. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  2. 2024An Adversarial Perspective on Machine Unlearning for AI Safetycited by 113
  3. 2025LLM Social Simulations Are a Promising Research Methodcited by 97
  4. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 93
  5. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  6. 2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 82
  7. 2024Finding Transformer Circuits with Edge Pruningcited by 58
  8. 2025Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometrycited by 58
  9. 2024What Makes and Breaks Safety Fine-tuning? A Mechanistic Studycited by 54
  10. 2024Language Models Resist Alignment: Evidence From Data Compressioncited by 50
  11. 2024ICLR: In-Context Learning of Representationscited by 49
  12. 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 45
  13. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  14. 2024Stress-Testing Capability Elicitation With Password-Locked Modelscited by 42
  15. 2024Model Merging and Safety Alignment: One Bad Model Spoils the Bunchcited by 41
  16. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 36
  17. 2025Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuningcited by 34
  18. 2024Immunization against harmful fine-tuning attackscited by 33
  19. 2024Towards Reliable Evaluation of Behavior Steering Interventions in LLMscited by 33
  20. 2025Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuningcited by 31
  21. 2024No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attackscited by 28
  22. 2023Exploiting Novel GPT-4 APIscited by 27
  23. 2025Narrow Finetuning Leaves Clearly Readable Traces in Activation Differencescited by 24
  24. 2024Understanding Reasoning Ability of Language Models From the Perspective of Reasoning Paths Aggregationcited by 22
  25. 2024Towards Uncovering How Large Language Model Works: An Explainability Perspectivecited by 22
  26. 2025Towards Understanding Fine-Tuning Mechanisms of LLMs via Circuit Analysiscited by 20
  27. 2024Bayesian scaling laws for in-context learningcited by 18
  28. 2024AI Risk Management Should Incorporate Both Safety and Securitycited by 17
  29. 2025Unlocking the Power of Function Vectors for Characterizing and Mitigating Catastrophic Forgetting in Continual Instruction Tuningcited by 16
  30. 2024On Catastrophic Inheritance of Large Foundation Modelscited by 15
  31. 2024Balancing Enhancement, Harmlessness, and General Capabilities: Enhancing Conversational LLMs with Direct RLHFcited by 15
  32. 2024Lessons from Studying Two-Hop Latent Reasoningcited by 15
  33. 2025The Elicitation Game: Evaluating Capability Elicitation Techniquescited by 15
  34. 2024SmileyLlama: Modifying Large Language Models for Directed Chemical Space Explorationcited by 14
  35. 2025Mechanistic Interpretability as Statistical Estimation: A Variance Analysiscited by 14
  36. 2024Near to Mid-term Risks and Opportunities of Open-Source Generative AIcited by 13
  37. 2025Is Random Attention Sufficient for Sequence Modeling? Disentangling Trainable Components in the Transformercited by 12
  38. 2025On the Impossibility of Separating Intelligence from Judgment: The Computational Intractability of Filtering for AI Alignmentcited by 12

50 most cited of 110 All 110 (JSON)