mlsanityData to 7 Oct 2026

Understanding Catastrophic Forgetting in Language Models via Implicit Inference

arXiv 2309.10105 ↗First posted 18 Sep 2023cs.CL cs.LGcited by 138

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification

Abstract (arXiv)

We lack a systematic understanding of the effects of fine-tuning (via methods such as instruction-tuning or reinforcement learning from human feedback), particularly on tasks outside the narrow fine-tuning distribution. In a simplified scenario, we demonstrate that improving performance on tasks within the fine-tuning data distribution comes at the expense of capabilities on other tasks. We hypothesize that language models implicitly infer the task of the prompt and that fine-tuning skews this inference towards tasks in the fine-tuning distribution. To test this, we propose Conjugate Prompting, which artificially makes the task look farther from the fine-tuning distribution while requiring the same capability, and we find that this recovers some of the pretraining capabilities in our synthetic setup. Since real-world fine-tuning distributions are predominantly English, we apply conjugate prompting to recover pretrained capabilities in LLMs by simply translating the prompts to different languages. This allows us to recover in-context learning abilities lost via instruction tuning, natural reasoning capability lost during code fine-tuning, and, more concerningly, harmful content generation suppressed by safety fine-tuning in chatbots like ChatGPT.

Key facts. Cited by 138 papers in the map (data through 7 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 48 of its 74 references that are in the map. First posted on arXiv 18 Sep 2023.

Builds on

48 of 74 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  9. 2017Deep reinforcement learning from human preferencescited by 3,846
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  12. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
36 more
  1. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  2. 2020Learning to summarize from human feedbackcited by 2,562
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  5. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  6. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  7. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  8. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  9. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  10. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  11. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  12. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  13. 2022Transformers learn in-context by gradient descentcited by 714
  14. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  15. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  16. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  17. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  18. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  19. 2016Learning to learn by gradient descent by gradient descentcited by 616
  20. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  21. 2021MetaICL: Learning to Learn In Contextcited by 450
  22. 2023Larger language models do in-context learning differentlycited by 386
  23. 2017Measuring Catastrophic Forgetting in Neural Networkscited by 359
  24. 2023Are aligned neural networks adversarially aligned?cited by 305
  25. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  26. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  27. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  28. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 233
  29. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 220
  30. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  31. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 149
  32. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 102
  33. 2019Meta-Learning without Memorizationcited by 92
  34. 2020Meta Learning Backpropagation And Improving Itcited by 49
  35. 2022Task Ambiguity in Humans and Language Modelscited by 27
  36. 2022Continual learning: a feature extraction formalization, an efficient algorithm, and fundamental obstructionscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 38; 2025: 44; 2026: 53.

138 papers.

Sort:

By area:

  1. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  2. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  3. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  4. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 153
  5. 2024Continual Learning for Large Language Models: A Surveycited by 144
  6. 2024Empowering Biomedical Discovery with AI Agentscited by 117
  7. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performancecited by 111
  8. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
  9. 2024Guardrail Baselines for Unlearning in LLMscited by 90
  10. 2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 82
  11. 2025Retaining by Doing: The Role of On-Policy Data in Mitigating Forgettingcited by 82
  12. 2024Refusal-Trained LLMs Are Easily Jailbroken As Browser Agentscited by 61
38 more
  1. 2024Controlling Language and Diffusion Models by Transporting Activationscited by 59
  2. 2025PolyGuard: A Multilingual Safety Moderation Tool for 17 Languagescited by 57
  3. 2024What Makes and Breaks Safety Fine-tuning? A Mechanistic Studycited by 54
  4. 2025The Best Instruction-Tuning Data are Those That Fitcited by 48
  5. 2024A Safe Harbor for AI Evaluation and Red Teamingcited by 46
  6. 2025Crosslingual Reasoning through Test-Time Scalingcited by 46
  7. 2024Personalized Large Language Modelscited by 44
  8. 2025Multi-Attribute Steering of Language Models via Targeted Interventioncited by 44
  9. 2024A Comprehensive Study of Multimodal Large Language Models for Image Quality Assessmentcited by 43
  10. 2024HtmlRAG: HTML is Better Than Plain Text for Modeling Retrieved Knowledge in RAG Systemscited by 37
  11. 2024Small Models, Big Insights: Leveraging Slim Proxy Models To Decide When and What to Retrieve for LLMscited by 34
  12. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 34
  13. 2025Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learningcited by 32
  14. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 31
  15. 2025HSplitLoRA: A Heterogeneous Split Parameter-Efficient Fine-Tuning Framework for Large Language Modelscited by 30
  16. 2025RL Fine-Tuning Heals OOD Forgetting in SFTcited by 27
  17. 2024Self-MoE: Towards Compositional Large Language Models with Self-Specialized Expertscited by 22
  18. 2024Unlocking the Potential of Model Merging for Low-Resource Languagescited by 19
  19. 2024Endless Jailbreaks with Bijection Learningcited by 18
  20. 2024PolygloToxicityPrompts: Multilingual Evaluation of Neural Toxic Degeneration in Large Language Modelscited by 16
  21. 2025Unlocking the Power of Function Vectors for Characterizing and Mitigating Catastrophic Forgetting in Continual Instruction Tuningcited by 16
  22. 2025Agent Data Protocol: Unifying Datasets for Diverse, Effective Fine-tuning of LLM Agentscited by 16
  23. 2025PosterO: Structuring Layout Trees to Enable Language Models in Generalized Content-Aware Layout Generationcited by 15
  24. 2025QFFT, Question-Free Fine-Tuning for Adaptive Reasoningcited by 15
  25. 2024LEMoE: Advanced Mixture of Experts Adaptor for Lifelong Model Editing of Large Language Modelscited by 14
  26. 2024MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuningcited by 13
  27. 2025On the Impossibility of Separating Intelligence from Judgment: The Computational Intractability of Filtering for AI Alignmentcited by 12
  28. 2026Replaying pre-training data improves fine-tuningcited by 12
  29. 2025R1-RE: Cross-Domain Relation Extraction with RLVRcited by 11
  30. 2025Activation Steering with a Feedback Controllercited by 11
  31. 2024Evaluating the Factuality of Large Language Models using Large-Scale Knowledge Graphscited by 10
  32. 2024RE-Adapt: Reverse Engineered Adaptation of Large Language Modelscited by 9
  33. 2024Context-Parametric Inversion: Why Instruction Finetuning Can Worsen Context Reliancecited by 9
  34. 2025Self-Jailbreaking: Language Models Can Reason Themselves Out of Safety Alignment After Benign Reasoning Trainingcited by 9
  35. 2024Demystifying Language Model Forgetting with Low-rank Example Associationscited by 8
  36. 2024Self-Data Distillation for Recovering Quality in Pruned Large Language Modelscited by 8
  37. 2024Continual Memorization of Factoids in Language Modelscited by 8
  38. 2025Mapping Post-Training Forgetting in Language Models at Scalecited by 8

50 most cited of 138 All 138 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Why Reinforcement Fine-Tuning Enables MLLMs Preserve Prior Knowledge Better: A Data Perspective
  2. 2024Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning
  3. 2026Mechanistic Analysis of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
  4. 2025Catastrophic Forgetting in LLMs: A Comparative Analysis Across Language Tasks
  5. 2025Mitigating Forgetting Between Supervised and Reinforcement Learning Yields Stronger Reasoners
  6. 2026Are Parameter-Efficient Fine-tuning Methods Really Different?
  7. 2025Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Training
  8. 2026Emergence of Context Characteristics Sensitivity in Large Language Models