mlsanityData to 7 Oct 2026

FIRE: Frobenius-Isometry Reinitialization for Balancing the Stability-Plasticity Tradeoff

arXiv 2602.08040 ↗First posted 8 Feb 2026cs.LG cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

Deep neural networks trained on nonstationary data must balance stability (i.e., retaining prior knowledge) and plasticity (i.e., adapting to new tasks). Standard reinitialization methods, which reinitialize weights toward their original values, are widely used but difficult to tune: conservative reinitializations fail to restore plasticity, while aggressive ones erase useful knowledge. We propose FIRE, a principled reinitialization method that explicitly balances the stability-plasticity tradeoff. FIRE quantifies stability through Squared Frobenius Error (SFE), measuring proximity to past weights, and plasticity through Deviation from Isometry (DfI), reflecting weight isotropy. The reinitialization point is obtained by solving a constrained optimization problem, minimizing SFE subject to DfI being zero, which is efficiently approximated by Newton-Schulz iteration. FIRE is evaluated on continual visual learning (CIFAR-10 with ResNet-18), language modeling (OpenWebText with GPT-0.1B), and reinforcement learning (HumanoidBench with SAC and Atari games with DQN). Across all domains, FIRE consistently outperforms both naive training without intervention and standard reinitialization methods, demonstrating effective balancing of the stability-plasticity tradeoff.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 53 of its 70 references that are in the map. First posted on arXiv 8 Feb 2026.

Builds on

53 of 70 references.

References · 53

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  7. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  8. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  9. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  10. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  11. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  12. 2016Progressive Neural Networkscited by 1,332
38 more
  1. 2018Experience Replay for Continual Learningcited by 1,074
  2. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  3. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  4. 2017Lifelong Learning with Dynamically Expandable Networkscited by 689
  5. 2019Online Continual Learning with Maximally Interfered Retrievalcited by 376
  6. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  7. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  8. 2020Supermasks in Superpositioncited by 223
  9. 2023Continual Pre-training of Language Modelscited by 194
  10. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  11. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 171
  12. 2023Understanding plasticity in neural networkscited by 170
  13. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  14. 2017Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practicecited by 132
  15. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 124
  16. 2021Continual World: A Robotic Benchmark For Continual Reinforcement Learningcited by 112
  17. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 82
  18. 2023Deep Reinforcement Learning with Plasticity Injectioncited by 71
  19. 2024Modular Duality in Deep Learningcited by 71
  20. 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learningcited by 70
  21. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularizationcited by 66
  22. 2023Maintaining Plasticity in Continual Learning via Regenerative Regularizationcited by 63
  23. 2023On the Stability-Plasticity Dilemma of Class-Incremental Learningcited by 55
  24. 2024Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learningcited by 49
  25. 2023DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimizationcited by 45
  26. 2021A study on the plasticity of neural networkscited by 44
  27. 2024Slow and Steady Wins the Race: Maintaining Plasticity with Hare and Tortoise Networkscited by 40
  28. 2022CLAD: A realistic Continual Learning benchmark for Autonomous Drivingcited by 35
  29. 2023Continual Learning as Computationally Constrained Reinforcement Learningcited by 35
  30. 2025Hyperspherical Normalization for Scalable Deep Reinforcement Learningcited by 35
  31. 2024Learning Continually by Spectral Regularizationcited by 34
  32. 2024Weight Clipping for Deep Continual and Reinforcement Learningcited by 29
  33. 2023Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stagescited by 27
  34. 2024Parseval Regularization for Continual Reinforcement Learningcited by 21
  35. 2024Plastic Learning with Deep Fourier Featurescited by 14
  36. 2024Self-Normalized Resets for Plasticity in Continual Learningcited by 11
  37. 2024Adaptive Regularization of Representation Rank as an Implicit Constraint of Bellman Equationcited by 7
  38. 2024DASH: Warm-Starting Neural Network Training in Stationary Settings without Loss of Plasticitycited by 7

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026Position: Collusion Risks Among AI Reasoning Agents Justify Certification Requirements for Making Market Decisionscited by 1
  2. 2026FOGO: Forgetting-aware Orthogonalization Optimizercited by 0
  3. 2026Beyond One-Size-Fits-All: Diagnosis-Driven Online Reinforcement Learning with Offline Priorscited by 0
  4. 2026V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Controlcited by 0
  5. 2026SingularClip: Preventing Spectral Collapse to Maintain Plasticity in Continual and Reinforcement Learningcited by 0
  6. 2026Beyond Forgetting: Diagnosing and Harnessing Shared Reasoning in Continual RLVRcited by 0
  7. 2026Generalized Residual Closure: General Learning Dynamics for Stability-Plasticity Compatibilitycited by 0