mlsanityData to 14 Oct 2026

Implicit biases in multitask and continual learning from a backward error analysis perspective

arXiv 2311.00235 ↗First posted 1 Nov 2023stat.ML cs.AI cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Using backward error analysis, we compute implicit training biases in multitask and continual learning settings for neural networks trained with stochastic gradient descent. In particular, we derive modified losses that are implicitly minimized during training. They have three terms: the original loss, accounting for convergence, an implicit flatness regularization term proportional to the learning rate, and a last term, the conflict term, which can theoretically be detrimental to both convergence and implicit regularization. In multitask, the conflict term is a well-known quantity, measuring the gradient alignment between the tasks, while in continual learning the conflict term is a new quantity in deep learning optimization, although a basic tool in differential geometry: The Lie bracket between the task gradients.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 30 of its 35 references that are in the map. First posted on arXiv 1 Nov 2023.

Builds on

30 of 35 references.

References · 30

  1. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  2. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  3. 2017A Survey on Multi-Task Learningcited by 840
  4. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 803
  5. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 184
  6. 2020Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Modelscited by 169
  7. 2020Implicit Gradient Regularizationcited by 146
  8. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 142
  9. 2021An Empirical Investigation of the Role of Pre-training in Lifelong Learningcited by 134
  10. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 132
  11. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  12. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
18 more
  1. 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 63
  2. 2020Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalizationcited by 62
  3. 2022Continual evaluation for lifelong learning: Identifying the stability gapcited by 56
  4. 2020Implicit Regularization in ReLU Networks with the Square Losscited by 45
  5. 2022Why neural networks find simple solutions: the many regularizers of geometric complexitycited by 39
  6. 2021Implicit Gradient Alignment in Distributed and Federated Learningcited by 29
  7. 2023On the Implicit Bias of Adamcited by 27
  8. 2023Implicit regularization in Heavy-ball momentum accelerated stochastic gradient descentcited by 25
  9. 2023A Comprehensive Empirical Evaluation on Online Continual Learningcited by 24
  10. 2020On dissipative symplectic integration with applications to gradient-based optimizationcited by 23
  11. 2021Sequential Reptile: Inter-Task Gradient Alignment for Multilingual Learningcited by 16
  12. 2021Discretization Drift in Two-Player Gamescited by 14
  13. 2022Toward Equation of Motion for Deep Neural Networks: Continuous-time Gradient Descent and Discretization Error Analysiscited by 13
  14. 2023On a continuous time model of gradient descent dynamics and instability in deep learningcited by 11
  15. 2023Fast Diffusion Probabilistic Model Sampling through the lens of Backward Error Analysiscited by 8
  16. 2021The Geometric Occam's Razor Implicit in Deep Learningcited by 7
  17. 2023Create and Find Flatness: Building Flat Training Spaces in Advance for Continual Learningcited by 3
  18. 2023Implicit regularisation in stochastic gradient descent: from single-objective to two-player gamescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 2.

4 papers.

  1. 2025Commute Your Domains: Trajectory Optimality Criterion for Multi-Domain Learningcited by 9
  2. 2026Optimizer Memory Makes Shuffle Order a First-Order Source of Fine-Tuning Noisecited by 7
  3. 2024Majority Kernels: An Approach to Leverage Big Model Dynamics for Efficient Small Model Trainingcited by 0
  4. 2026Intrinsic-Extrinsic Coupling in Learning Dynamicscited by 0