# Conditions for Catastrophic Forgetting in Multilingual Translation

arXiv: https://arxiv.org/abs/2510.19546 · HTML: https://mlsanity.com/paper/2510.19546/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 50 references that are in the map.
- First posted on arXiv 22 Oct 2025.

## Builds on (most cited first, 29 of 29)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [LoRA Learns Less and Forgets Less](https://mlsanity.com/paper/2405.09673/) (2024)
- [Tower: An Open Multilingual Large Language Model for Translation-Related Tasks](https://mlsanity.com/paper/2402.17733/) (2024)
- [When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method](https://mlsanity.com/paper/2402.17193/) (2024)
- [Contrastive Learning for Many-to-many Multilingual Neural Machine Translation](https://mlsanity.com/paper/2105.09501/) (2021)
- [SONAR: Sentence-Level Multimodal and Language-Agnostic Representations](https://mlsanity.com/paper/2308.11466/) (2023)
- [Understanding and Mitigating Language Confusion in LLMs](https://mlsanity.com/paper/2406.20052/) (2024)
- [Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation](https://mlsanity.com/paper/2205.12647/) (2022)
- [Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations](https://mlsanity.com/paper/1906.01181/) (2019)
- [The Missing Ingredient in Zero-Shot Neural Machine Translation](https://mlsanity.com/paper/1903.07091/) (2019)
- [Steering Large Language Models for Machine Translation with Finetuning and In-Context Learning](https://mlsanity.com/paper/2310.13448/) (2023)
- [Improving Zero-shot Translation with Language-Independent Constraints](https://mlsanity.com/paper/1906.08584/) (2019)
- [Is Translation All You Need? A Study on Solving Multilingual Tasks with Large Language Models](https://mlsanity.com/paper/2403.10258/) (2024)
- [Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMs](https://mlsanity.com/paper/2502.14830/) (2025)
- [Overcoming Catastrophic Forgetting in Massively Multilingual Continual Learning](https://mlsanity.com/paper/2305.16252/) (2023)
- [Beyond English: The Impact of Prompt Translation Strategies across Languages and Tasks in Multilingual LLMs](https://mlsanity.com/paper/2502.09331/) (2025)
- [Investigating Catastrophic Forgetting During Continual Training for Neural Machine Translation](https://mlsanity.com/paper/2011.00678/) (2020)
- [Continual Learning in Multilingual NMT via Language-Specific Embeddings](https://mlsanity.com/paper/2110.10478/) (2021)
- [Large Language Model for Multi-Domain Translation: Benchmarking and Domain CoT Fine-tuning](https://mlsanity.com/paper/2410.02631/) (2024)
- [How Multilingual Are Large Language Models Fine-Tuned for Translation?](https://mlsanity.com/paper/2405.20512/) (2024)
- [Learning an Artificial Language for Knowledge-Sharing in Multilingual Translation](https://mlsanity.com/paper/2211.01292/) (2022)
- [Extending Multilingual Machine Translation through Imitation Learning](https://mlsanity.com/paper/2311.08538/) (2023)
- [Building Accurate Translation-Tailored LLMs with Language Aware Instruction Tuning](https://mlsanity.com/paper/2403.14399/) (2024)
- [Efficiently Upgrading Multilingual Machine Translation Models to Support More Languages](https://mlsanity.com/paper/2302.03528/) (2023)
- [Domain adapted machine translation: What does catastrophic forgetting forget and why?](https://mlsanity.com/paper/2412.17537/) (2024)

## Built on by (most cited first, 10 of 10)

- [Robust Policy Optimization to Prevent Catastrophic Forgetting](https://mlsanity.com/paper/2602.08813/) (2026)
- [CapTrack: Multifaceted Evaluation of Forgetting in LLM Post-Training](https://mlsanity.com/paper/2603.06610/) (2026)
- [Reinforcement Learning with Semantic Rewards Enables Low-Resource Language Expansion without Alignment Tax](https://mlsanity.com/paper/2605.14366/) (2026)
- [Shaping Schema via Language Representation as the Next Frontier for LLM Intelligence Expanding](https://mlsanity.com/paper/2605.09271/) (2026)
- [Just Use XML: Revisiting Joint Translation and Label Projection](https://mlsanity.com/paper/2603.12021/) (2026)
- [Cross-Lingual Transfer for Machine Translation in Turkic Languages](https://mlsanity.com/paper/2607.29355/) (2026)
- [Bridging the English-Arabic Medical Knowledge Gap: Targeted Low-Rank Adaptation via Causal Layer Selection](https://mlsanity.com/paper/2608.00207/) (2026)
- [GRPO Beyond English: A Large-Scale Study of GRPO in Non-English and Multilingual Settings](https://mlsanity.com/paper/2608.13698/) (2026)
- [Fine-Tuning LLMs for Translation: General Forgetting Mitigation Does Not Preserve MT-Specific Instruction Following](https://mlsanity.com/paper/2609.28395/) (2026)
- [Identifying Temporal Features within Transcoders for Time Sensitive Factual Recall](https://mlsanity.com/paper/2609.33183/) (2026)

## Abstract (arXiv)

Fine-tuning multilingual foundation models on specific languages often induces catastrophic forgetting, degrading performance on languages unseen in fine-tuning. While this phenomenon is widely-documented, the literature presents fragmented results about when forgetting occurs. To address this ambiguity, we conduct a systematic empirical study using machine translation as a testbed to identify the conditions that trigger catastrophic forgetting in multilingual fine-tuning. Through controlled experiments across different model architectures, data scales, and fine-tuning approaches, we reveal that the relative scale between model and data size is a primary determinant of forgetting. Moreover, we demonstrate that a model's instruction-following ability is more critical for retaining multilingual knowledge than its architecture. Contrary to assumptions, parameter-efficient fine-tuning offers no clear advantage over full fine-tuning in mitigating forgetting. Lastly, we show that cross-lingual alignment can mitigate forgetting while also facilitating positive transfer to unseen target languages.