mlsanityData to 7 Oct 2026

Hidden Failure Modes of Gradient Modification under Adam in Continual Learning, and Adaptive Decoupled Moment Routing as a Repair

arXiv 2604.22407 ↗First posted 24 Apr 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Many continual-learning methods modify gradients upstream (e.g., projection, penalty rescaling, replay mixing) while treating Adam as a neutral backend. We show this composition has a hidden failure mode. In a high-overlap, non-adaptive 8-domain continual LM, all shared-routing projection baselines collapse close to vanilla forgetting (12.5--12.8 vs. 13.2). A 0.5% replay buffer is the strongest shared alternative but still reaches 11.6, while fixed-strength decoupling falls below vanilla at 14.1. Only adaptive decoupled routing remains stable at 9.4, improving over vanilla by 3.8 units. On a 16-domain stream, its gain over the strongest shared-routing projection baseline grows to 4.5--4.8 units. The failure is largely invisible on clean benchmarks. We explain this effect through Adam's second-moment pathway: in the tested regime, projection induces a 1/(1-alpha) inflation of the old-direction effective learning rate, matching measurements within 8% across eight alpha values. The same conflict appears with penalty methods, replay mixing, and at 7B scale under LoRA. Our fix routes the modified gradient only to the first moment while preserving magnitude-faithful second-moment statistics, with overlap-aware adaptive strength. This simple change is the only tested configuration that consistently avoids collapse across methods, optimizers, and scale.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 32 of its 41 references that are in the map. First posted on arXiv 24 Apr 2026.

Builds on

32 of 41 references.

References · 32

  1. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  3. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  4. 2017Continual Learning with Deep Generative Replaycited by 1,152
  5. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  6. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  7. 2019On the Convergence of Adam and Beyondcited by 834
  8. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 511
  9. 2019Orthogonal Gradient Descent for Continual Learningcited by 361
  10. 2021Gradient Projection Memory for Continual Learningcited by 338
  11. 2024InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learningcited by 189
  12. 2019Adaptive Gradient Methods with Dynamic Bound of Learning Ratecited by 181
20 more
  1. 2018Continual Learning of Context-dependent Processing in Neural Networkscited by 178
  2. 2020Understanding the Role of Training Regimes in Continual Learningcited by 177
  3. 2021Training Networks in Null Space of Feature Covariance for Continual Learningcited by 139
  4. 2022TRGP: Trust Region Gradient Projection for Continual Learningcited by 79
  5. 2021Layerwise Optimization by Gradient Decomposition for Continual Learningcited by 51
  6. 2024Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learningcited by 49
  7. 2023Continual Learning with Scaled Gradient Projectioncited by 33
  8. 2022Continual Learning with Recursive Gradient Optimizationcited by 32
  9. 2023Two Complementary Perspectives to Continual Learning: Ask Not Only What to Optimize, But Also Howcited by 16
  10. 2025Sculpting Subspaces: Constrained Full Fine-Tuning in LLMs for Continual Learningcited by 15
  11. 2024MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuningcited by 13
  12. 2019An Adaptive and Momental Bound Method for Stochastic Learningcited by 11
  13. 2023Class Gradient Projection For Continual Learningcited by 11
  14. 2025Continual Gradient Low-Rank Projection Fine-Tuning for LLMscited by 10
  15. 2025Continuous Subspace Optimization for Continual Learningcited by 8
  16. 2025SplitLoRA: Balancing Stability and Plasticity in Continual Learning Through Gradient Space Splittingcited by 7
  17. 2023Restricted Orthogonal Gradient Projection for Continual Learningcited by 6
  18. 2026Beyond Static Models: An Evolving Framework for Continual Learning in Large Language Models across Training Stagescited by 4
  19. 2019Continual Learning in Deep Neural Network by Using a Kalman Optimisercited by 1
  20. 2026Fisher-Orthogonal Projected Natural Gradient Descent for Continual Learningcited by 1

Built on by

Not cited yet.