mlsanityData to 14 Oct 2026

Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuning

arXiv 2507.16795 ↗First posted 22 Jul 2025cs.LG cs.AI cs.CLcited by 39

Not on the map. Why?

Abstract (arXiv)

Fine-tuning large language models (LLMs) can lead to unintended out-of-distribution generalization. Standard approaches to this problem rely on modifying training data, for example by adding data that better specify the intended generalization. However, this is not always practical. We introduce Concept Ablation Fine-Tuning (CAFT), a technique that leverages interpretability tools to control how LLMs generalize from fine-tuning, without needing to modify the training data or otherwise use data from the target distribution. Given a set of directions in an LLM's latent space corresponding to undesired concepts, CAFT works by ablating these concepts with linear projections during fine-tuning, steering the model away from unintended generalizations. We successfully apply CAFT to three fine-tuning tasks, including emergent misalignment, a phenomenon where LLMs fine-tuned on a narrow task generalize to give egregiously misaligned responses to general questions. Without any changes to the fine-tuning data, CAFT reduces misaligned responses by 10x without degrading performance on the training distribution. Overall, CAFT represents a novel approach for steering LLM generalization without modifying training data.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Builds on 60 of its 90 references that are in the map. First posted on arXiv 22 Jul 2025.

Builds on

60 of 90 references.

References · 60

  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  2. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  3. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  4. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  5. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  6. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  7. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,312
  8. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  9. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  10. 2022Mass-Editing Memory in a Transformercited by 1,001
  11. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  12. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
38 more
  1. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  2. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  3. 2021Editing Factual Knowledge in Language Modelscited by 553
  4. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 523
  5. 2021Fast Model Editing at Scalecited by 498
  6. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  7. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 455
  8. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 435
  9. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 394
  10. 2018Confounding variables can degrade generalization performance of radiological deep learning modelscited by 379
  11. 2019Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Settingcited by 349
  12. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 334
  13. 2023Studying Large Language Model Generalization with Influence Functionscited by 326
  14. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 319
  15. 2023Function Vectors in Large Language Modelscited by 308
  16. 2023LEACE: Perfect linear concept erasure in closed formcited by 267
  17. 2018Learning Not to Learn: Training Deep Neural Networks with Biased Datacited by 255
  18. 2023ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversationcited by 231
  19. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 225
  20. 2025AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoderscited by 218
  21. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 172
  22. 2024Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Trackingcited by 149
  23. 2019Distributionally Robust Language Modelingcited by 143
  24. 2017Controllable Invariance through Adversarial Feature Learningcited by 127
  25. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 126
  26. 2020Towards Debiasing NLU Models from Unknown Biasescited by 116
  27. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 115
  28. 2025Are Sparse Autoencoders Useful? A Case Study in Sparse Probingcited by 115
  29. 2024Automatically Interpreting Millions of Features in Large Language Modelscited by 113
  30. 2022Linear Adversarial Concept Erasurecited by 101
  31. 2024On scalable oversight with weak LLMs judging strong LLMscited by 99
  32. 2024Guardrail Baselines for Unlearning in LLMscited by 96
  33. 2025Model Organisms for Emergent Misalignmentcited by 91
  34. 2022Spread Spurious Attribute: Improving Worst-group Accuracy with Spurious Attribute Estimationcited by 88
  35. 2020Coping with Label Shift via Distributionally Robust Optimisationcited by 70
  36. 2022MaskTune: Mitigating Spurious Correlations by Forcing to Explorecited by 66
  37. 2024Applying sparse autoencoders to unlearn knowledge in language modelscited by 59
  38. 2022Kernelized Concept Erasurecited by 48

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 31.

39 papers.

Sort:

By area:

  1. 2025Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-timecited by 43
  2. 2025Narrow Finetuning Leaves Clearly Readable Traces in Activation Differencescited by 28
  3. 2026Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Modelscited by 21
  4. 2025Steering Language Models with Weight Arithmeticcited by 20
  5. 2026Model Spec Midtraining: Improving How Alignment Training Generalizescited by 20
  6. 2025Spilling the Beans: Teaching LLMs to Self-Report Their Hidden Objectivescited by 15
  7. 2025Hallucination reduction with CASAL: Contrastive Activation Steering For Amortized Learningcited by 6
  8. 2026Narrow Fine-Tuning Erodes Safety Alignment in Vision-Language Agentscited by 6
  9. 2026Delta-Crosscoder: Robust Crosscoder Model Diffing in Narrow Fine-Tuning Regimescited by 4
  10. 2025Detecting Adversarial Fine-tuning with Auditing Agentscited by 3
  11. 2026BLOCK-EM: Preventing Emergent Misalignment via Latent Blockingcited by 3
  12. 2026One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Modelscited by 3
27 more
  1. 2026Reinforcement Learning Can Amplify Emergent Misalignment from Harmless Rewardscited by 3
  2. 2026Inoculation Adapters: Improved Selective Generalization of Capabilities with Fewer Surprising Backdoorscited by 3
  3. 2026Emergent Misalignment Recruits a Pre-existing Persona Subspacecited by 3
  4. 2025The Unintended Trade-off of AI Alignment:Balancing Hallucination Mitigation and Safety in LLMscited by 2
  5. 2025Can SAEs reveal and mitigate racial biases of LLMs in healthcare?cited by 2
  6. 2026Convergent World Representations and Divergent Taskscited by 2
  7. 2026Overtrained, Not Misalignedcited by 2
  8. 2026Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signalcited by 2
  9. 2026Objective Matters: Fine-Tuning Objectives Shape Safety, Robustness, and Persona Driftcited by 1
  10. 2026Not Just the Destination, But the Journey: Reasoning Traces Causally Shape Generalization Behaviorscited by 1
  11. 2026Actionable Activation Directions for Detecting and Mitigating Emergent Misalignment Across Language Model Familiescited by 1
  12. 2026Out-of-Distribution Generalization of Risk Aversion in Language Modelscited by 1
  13. 2026From Weights to Words: Expressing and Editing Preference Model Inferences in Natural Languagecited by 1
  14. 2026Inference-Time Consensus for Mitigating Hidden Behaviors from LLM Fine-Tuningcited by 1
  15. 2026Data Attribution of Emergent Misalignment with Persona Featurescited by 1
  16. 2026The Unequal Influence of Bad Advice: Using Training Data Attribution to Modulate Emergent Misalignmentcited by 1
  17. 2026MoRFI: Monotonic Sparse Autoencoder Feature Identificationcited by 0
  18. 2026Trait-space Monitoring for Emergent Misalignment During Supervised Finetuningcited by 0
  19. 2026Shared Latent Structures Enable Unified Backdoor Detection and Mitigation in LLMscited by 0
  20. 2026Transplanting, inverting, and preventing a misalignment persona: method-conditional emergent misalignment in Qwen2.5cited by 0
  21. 2026The Storyteller in the Model: Narrative Pattern Inheritance, Escalation Dynamics, and Alignment Governance in LLMscited by 0
  22. 2026FOCUS: Decoupling Expert Personas in LLMs to Enhance Domain Expert Capabilitiescited by 0
  23. 2026Making Clinical Language Models Auditable: Concept-Guided Fine-Tuning for Robust Predictioncited by 0
  24. 2026Probe Generalization as Subspace Selection for OOD Deception Detectioncited by 0
  25. 2026Less Sycophancy, Stronger Refusal? Lessons for AI Safety from Mechanistic Interpretabilitycited by 0
  26. 2026Learning a Fact Is Not Learning How to Retrieve Itcited by 0
  27. 2026Predicting how model propensities generalize out of distributioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Selective Self-to-Supervised Fine-Tuning for Generalization in Large Language Models
  2. 2026Diversity in Large Language Models under Supervised Fine-Tuning
  3. 2025Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment Quality
  4. 2025On the generalization of language models from in-context learning and finetuning: a controlled study
  5. 2024TAIA: Large Language Models are Out-of-Distribution Data Learners
  6. 2025PAFT: Prompt-Agnostic Fine-Tuning
  7. 2025Complexity-aware fine-tuning
  8. 2025Efficient Response Generation Strategy Selection for Fine-Tuning Large Language Models Through Self-Aligned Perplexity