# A Deafening Silence: Catastrophic Forgetting Lives in the Output Embeddings of Tokens the Data Never Speaks

arXiv: https://arxiv.org/abs/2610.09835 · HTML: https://mlsanity.com/paper/2610.09835/

## Key facts

- No citing papers in the map yet (data through 7 Oct 2026).
- Builds on 19 of its 28 references that are in the map.
- First posted on arXiv 7 Oct 2026.

## Builds on (most cited first, 19 of 19)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](https://mlsanity.com/paper/2012.13255/) (2020)
- [Class-incremental learning: survey and performance evaluation on image classification](https://mlsanity.com/paper/2010.15277/) (2020)
- [Gradient Projection Memory for Continual Learning](https://mlsanity.com/paper/2103.09762/) (2021)
- [LoRA Learns Less and Forgets Less](https://mlsanity.com/paper/2405.09673/) (2024)
- [Orthogonal Subspace Learning for Language Model Continual Learning](https://mlsanity.com/paper/2310.14152/) (2023)
- [On Empirical Comparisons of Optimizers for Deep Learning](https://mlsanity.com/paper/1910.05446/) (2019)
- [Spurious Forgetting in Continual Learning of Language Models](https://mlsanity.com/paper/2501.13453/) (2025)
- [Tensor Programs IVb: Adaptive Optimization in the Infinite-Width Limit](https://mlsanity.com/paper/2308.01814/) (2023)
- [Investigating Catastrophic Forgetting During Continual Training for Neural Machine Translation](https://mlsanity.com/paper/2011.00678/) (2020)
- [MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuning](https://mlsanity.com/paper/2407.20999/) (2024)
- [Mechanistic Analysis of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning](https://mlsanity.com/paper/2601.18699/) (2026)
- [Continual Learning Optimizations for Auto-regressive Decoder of Multilingual ASR systems](https://mlsanity.com/paper/2407.03645/) (2024)
- [Scalable Strategies for Continual Learning with Replay](https://mlsanity.com/paper/2505.12512/) (2025)
- [Catastrophic Forgetting is Low-Rank: A Function-Space Theory for Continual Adaptation](https://mlsanity.com/paper/2606.18024/) (2026)

## Abstract (arXiv)

Continual pre-training and fine-tuning in Large Language Models (LLMs) inevitably induce catastrophic forgetting, typically mitigated by replay using often-inaccessible original data. In this data-free regime, we analyze where forgetting occurs and why. Systematic parameter freezing across five settings up to 1.4B reveals that forgetting concentrates selectively in the output embeddings of tokens rarely seen in the new corpus, whereas the same sqrt(v-hat) band of the body is inert and new learning resides elsewhere. This localization is governed by the vocabulary deficiency of the corpus rather than the training mode, allowing pre-retraining risk ranking from token counts alone within a fixed base model. Mechanistically, absent tokens receive persistent one-sided softmax gradients that Adam's second-moment (sqrt(v-hat)) normalization amplifies into full-sized updates. We therefore propose an intervention: raising Adam's epsilon exclusively for the output projection during training. Across eight settings spanning 160M to 12B parameters and four model families, this removes 39.4% to 67.9% of forgetting across all seven stable configurations without degrading target learning or requiring per-model tuning. The defense combines additively or better with replay (79.8% on Qwen/Korean) and rescues released-head LoRA from a 23-fold forgetting surge. Because post-hoc editing of the drifted rows recovers under 5% of forgetting, the intervention must operate during training. Our findings indicate that a single-line optimizer adjustment may serve as the primary defense against catastrophic forgetting where the corpus starves the vocabulary.