mlsanityData to 14 Oct 2026

Sharpness-Aware Pretraining Mitigates Catastrophic Forgetting

arXiv 2605.02105 ↗First posted 4 May 2026cs.LG cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Pretraining optimizers are tuned to produce the strongest possible base model, on the assumption that a stronger starting point yields a stronger model after subsequent changes like post-training and quantization. This overlooks the geometry of the base model which controls how much of the base model's capabilities survive subsequent parameter updates. We study three pretraining optimization approaches that bias optimization toward flatter minima: Sharpness-Aware Minimization (SAM), large learning rates, and shortened learning rate annealing periods. Across model sizes ranging from 20M to 150M parameters, we find that these interventions consistently improve downstream performance after post-training on five common datasets with up to 80% less forgetting. These principles hold at scale: a short SAM mid-training phase applied to an existing OLMo-2-1B checkpoint reduces forgetting by 31% after MetaMath post-training and by 40% after 4-bit quantization.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 52 of its 62 references that are in the map. First posted on arXiv 4 May 2026.

Builds on

52 of 62 references.

References · 52

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  8. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  9. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  10. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  11. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  12. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
38 more
  1. 2016Progressive Neural Networkscited by 1,361
  2. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  3. 2017Continual Learning with Deep Generative Replaycited by 1,158
  4. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,105
  5. 2018Efficient Lifelong Learning with A-GEMcited by 1,058
  6. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  7. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  8. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  9. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
  10. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  11. 2021Editing Factual Knowledge in Language Modelscited by 553
  12. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  13. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  14. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  15. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
  16. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 359
  17. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  18. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  19. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 172
  20. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  21. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
  22. 2022A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learningcited by 154
  23. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 144
  24. 2021An Empirical Investigation of the Role of Pre-training in Lifelong Learningcited by 134
  25. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 92
  26. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  27. 2024Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspectivecited by 81
  28. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 68
  29. 2022How Does Sharpness-Aware Minimization Minimize Sharpness?cited by 65
  30. 2025Overtrained Language Models Are Harder to Fine-Tunecited by 58
  31. 2019Shaping the learning landscape in neural networks around wide flat minimacited by 47
  32. 2024Scaling Optimal LR Across Token Horizonscited by 40
  33. 2023Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalizationcited by 39
  34. 2024Make Continual Learning Stronger via C-Flatcited by 35
  35. 2024Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Trainingcited by 24
  36. 2025Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churncited by 22
  37. 2024Editing Conceptual Knowledge for Large Language Modelscited by 21
  38. 2025Training Dynamics Impact Post-Training Quantization Robustnesscited by 15

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replaycited by 7
  2. 2026Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectorscited by 3
  3. 2026TailSFT: Filtered Fine-Tuning Improves Post-Training Performancecited by 2
  4. 2026Double Preconditioning (DoPr): Optimization for Test-Time Performance, not Validation Losscited by 1
  5. 2026Preference Data Selection for Mitigating the Alignment Tax in Large Language Modelscited by 0
  6. 2026Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recallcited by 0
  7. 2026Good Pretraining, Bad SFT: Checkpoint Quality Across the Training Stackcited by 0
  8. 2026Linger and Lose: Knowledge Collapse in Low-Bit Language Modelscited by 0
  9. 2026Understanding the Weight Averaging Mechanism in LLM Training for Post-Training Quantizationcited by 0
  10. 2026Off-Policy Merging Beats On-Policy Self-Distillation for Continual Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Sharpness-Aware Machine Unlearning
  2. 2026Understanding Quantization of Optimizer States in LLM Pre-training: Dynamics of State Staleness and Effectiveness of State Resets
  3. 2026CapTrack: Multifaceted Evaluation of Forgetting in LLM Post-Training
  4. 2026SAUL: Sharpness-Aware Augmented-Lagrangian Unlearning
  5. 2026Quantization-Robust Unlearning through the Lens of Retain-Forget Loss Landscapes Interaction
  6. 2026FlashOptim: Optimizers for Memory-Efficient Training
  7. 2026Mechanistic Analysis of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
  8. 2026On the Plasticity Collapse in Continual Machine Unlearning