mlsanityData to 7 Oct 2026

MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuning

arXiv 2407.20999 ↗First posted 30 Jul 2024cs.LG cs.AIcited by 13

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks. Typically, LLMs are first pre-trained on large corpora and subsequently fine-tuned on task-specific datasets. However, during fine-tuning, LLMs may forget some knowledge acquired in the pre-training stage, leading to a decline in general capabilities. Existing approaches to mitigate forgetting often rely on access to pre-training data, which may be unavailable in many real-world scenarios--such as fine-tuning checkpoint-only open-source LLMs. To address this challenge, we propose a new fine-tuning algorithm termed Momentum-Filtered Optimizer (MoFO). MoFO is an extension of greedy block coordinate descent (BCD) methods: in each iteration, MoFO only updates the model parameters with the largest momentum magnitudes, while keeping all other parameters fixed. MoFO achieves similar fine-tuning performance to the default fine-tuning algorithm while effectively mitigating knowledge forgetting. We validate MoFO through rigorous convergence analysis and extensive experiments, demonstrating its effectiveness in mitigating forgetting without pre-training data.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 92 of its 115 references that are in the map. First posted on arXiv 30 Jul 2024.

Builds on

92 of 115 references.

References · 92

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2016Learning without Forgettingcited by 2,465
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  12. 2023Code Llama: Open Foundation Models for Codecited by 2,105
38 more
  1. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  2. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  3. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  4. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  5. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  6. 2022Editing Models with Task Arithmeticcited by 1,076
  7. 2018Experience Replay for Continual Learningcited by 1,074
  8. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  9. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 895
  10. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  11. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  12. 2021Robust fine-tuning of zero-shot modelscited by 785
  13. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
  14. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  15. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  16. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  17. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  18. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  19. 2019Gradient based sample selection for online continual learningcited by 568
  20. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
  21. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 511
  22. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  23. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  24. 2019Online Continual Learning with Maximally Interfered Retrievalcited by 376
  25. 2019Orthogonal Gradient Descent for Continual Learningcited by 361
  26. 2017Measuring Catastrophic Forgetting in Neural Networkscited by 359
  27. 2024LoRA Learns Less and Forgets Lesscited by 313
  28. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 296
  29. 2021Co2L: Contrastive Continual Learningcited by 243
  30. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 231
  31. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  32. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227
  33. 2018Explicit Inductive Bias for Transfer Learning with Convolutional Networkscited by 226
  34. 2015Semi-supervised Sequence Learningcited by 198
  35. 2018Online Structured Laplace Approximations For Overcoming Catastrophic Forgettingcited by 197
  36. 2023Progressive Prompts: Continual Learning for Language Modelscited by 195
  37. 2023Continual Pre-training of Language Modelscited by 194
  38. 2019Meta-Learning Representations for Continual Learningcited by 189

50 most cited of 92 All 92 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 6.

13 papers.

Sort:

By area:

  1. 2024LeanAgent: Lifelong Learning for Formal Theorem Provingcited by 34
  2. 2025Upweighting Easy Samples in Fine-Tuning Mitigates Forgettingcited by 16
  3. 2025A Unified Gradient-based Framework for Task-agnostic Continual Learning-Unlearningcited by 6
  4. 2025Context-Free Synthetic Data Mitigates Forgettingcited by 5
  5. 2025T2I-ConBench: Text-to-Image Benchmark for Continual Post-trainingcited by 1
  6. 2025Logits Replay + MoClip: Stabilized, Low-Cost Post-Training with Minimal Forgettingcited by 1
  7. 2026Collaborative Parameter Learning: Mitigating Forgetting via Parameter-Level Gradient Analysiscited by 1
  8. 2026Symbiotic-MoE: Unlocking the Synergy between Generation and Understandingcited by 1
  9. 2025PTPP-Aware Adaptation Scaling Laws: Predicting Domain-Adaptation Performance at Unseen Pre-Training Budgetscited by 0
  10. 2026Hidden Failure Modes of Gradient Modification under Adam in Continual Learning, and Adaptive Decoupled Moment Routing as a Repaircited by 0
  11. 2026Memory as a Markov Matrix: Sample Efficient Knowledge Expansion via Token-to-Dictionary Mappingcited by 0
  12. 2026One Algorithm, Two Goals: Dual Scoring for Parameter and Data Selection in LLM Fine-Tuningcited by 0
1 more
  1. 2026A Deafening Silence: Catastrophic Forgetting Lives in the Output Embeddings of Tokens the Data Never Speakscited by 0