# Catastrophic Forgetting in LLMs: A Comparative Analysis Across Language Tasks

arXiv: https://arxiv.org/abs/2504.01241 · HTML: https://mlsanity.com/paper/2504.01241/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 16 references that are in the map.
- First posted on arXiv 1 Apr 2025.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning](https://mlsanity.com/paper/2308.08747/) (2023)
- [Measuring Catastrophic Forgetting in Neural Networks](https://mlsanity.com/paper/1708.02072/) (2017)
- [Orca 2: Teaching Small Language Models How to Reason](https://mlsanity.com/paper/2311.11045/) (2023)
- [Fine-tuned Language Models are Continual Learners](https://mlsanity.com/paper/2205.12393/) (2022)
- [CURLoRA: Stable LLM Continual Fine-Tuning and Catastrophic Forgetting Mitigation](https://mlsanity.com/paper/2408.14572/) (2024)

## Built on by (most cited first, 8 of 8)

- [Mapping Post-Training Forgetting in Language Models at Scale](https://mlsanity.com/paper/2510.17776/) (2025)
- [Safety in Self-Evolving LLM Agent Systems: Threats, Amplification, and Case Studies](https://mlsanity.com/paper/2606.23075/) (2026)
- [How Much is Too Much? Exploring LoRA Rank Trade-offs for Retaining Knowledge and Domain Robustness](https://mlsanity.com/paper/2512.15634/) (2025)
- [CapTrack: Multifaceted Evaluation of Forgetting in LLM Post-Training](https://mlsanity.com/paper/2603.06610/) (2026)
- [Poison Once, Refuse Forever: Weaponizing Alignment for Injecting Bias in LLMs](https://mlsanity.com/paper/2508.20333/) (2025)
- [MixtureKit: A General Framework for Composing, Training, and Visualizing Mixture-of-Experts Models](https://mlsanity.com/paper/2512.12121/) (2025)
- [Memory as a Markov Matrix: Sample Efficient Knowledge Expansion via Token-to-Dictionary Mapping](https://mlsanity.com/paper/2605.04308/) (2026)
- [Building Socio-Affective Artificial Intelligence for Interactive Multi-Agent Simulations](https://mlsanity.com/paper/2609.26927/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have significantly advanced Natural Language Processing (NLP), particularly in Natural Language Understanding (NLU) tasks. As we progress toward an agentic world where LLM-based agents autonomously handle specialized tasks, it becomes crucial for these models to adapt to new tasks without forgetting previously learned information - a challenge known as catastrophic forgetting. This study evaluates the continual fine-tuning of various open-source LLMs with different parameter sizes (specifically models under 10 billion parameters) on key NLU tasks from the GLUE benchmark, including SST-2, MRPC, CoLA, and MNLI. By employing prompt engineering and task-specific adjustments, we assess and compare the models' abilities to retain prior knowledge while learning new tasks. Our results indicate that models such as Phi-3.5-mini exhibit minimal forgetting while maintaining strong learning capabilities, making them well-suited for continual learning environments. Additionally, models like Orca-2-7b and Qwen2.5-7B demonstrate impressive learning abilities and overall performance after fine-tuning. This work contributes to understanding catastrophic forgetting in LLMs and highlights prompting engineering to optimize model performance for continual learning scenarios.