# Implicit biases in multitask and continual learning from a backward error analysis perspective

arXiv: https://arxiv.org/abs/2311.00235 · HTML: https://mlsanity.com/paper/2311.00235/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 35 references that are in the map.
- First posted on arXiv 1 Nov 2023.

## Builds on (most cited first, 30 of 30)

- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [On the Origin of Implicit Regularization in Stochastic Gradient Descent](https://mlsanity.com/paper/2101.12176/) (2021)
- [Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models](https://mlsanity.com/paper/2010.05874/) (2020)
- [Implicit Gradient Regularization](https://mlsanity.com/paper/2009.11162/) (2020)
- [Stochastic modified equations and adaptive stochastic gradient algorithms](https://mlsanity.com/paper/1511.06251/) (2015)
- [An Empirical Investigation of the Role of Pre-training in Lifelong Learning](https://mlsanity.com/paper/2112.09153/) (2021)
- [Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation](https://mlsanity.com/paper/2105.14368/) (2021)
- [Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like process](https://mlsanity.com/paper/1904.09080/) (2019)
- [Label Noise SGD Provably Prefers Flat Global Minimizers](https://mlsanity.com/paper/2106.06530/) (2021)
- [On Linear Stability of SGD and Input-Smoothness of Neural Networks](https://mlsanity.com/paper/2105.13462/) (2021)
- [Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalization](https://mlsanity.com/paper/2012.14193/) (2020)
- [Continual evaluation for lifelong learning: Identifying the stability gap](https://mlsanity.com/paper/2205.13452/) (2022)
- [Implicit Regularization in ReLU Networks with the Square Loss](https://mlsanity.com/paper/2012.05156/) (2020)
- [Why neural networks find simple solutions: the many regularizers of geometric complexity](https://mlsanity.com/paper/2209.13083/) (2022)
- [Implicit Gradient Alignment in Distributed and Federated Learning](https://mlsanity.com/paper/2106.13897/) (2021)
- [On the Implicit Bias of Adam](https://mlsanity.com/paper/2309.00079/) (2023)
- [Implicit regularization in Heavy-ball momentum accelerated stochastic gradient descent](https://mlsanity.com/paper/2302.00849/) (2023)
- [A Comprehensive Empirical Evaluation on Online Continual Learning](https://mlsanity.com/paper/2308.10328/) (2023)
- [On dissipative symplectic integration with applications to gradient-based optimization](https://mlsanity.com/paper/2004.06840/) (2020)
- [Sequential Reptile: Inter-Task Gradient Alignment for Multilingual Learning](https://mlsanity.com/paper/2110.02600/) (2021)
- [Discretization Drift in Two-Player Games](https://mlsanity.com/paper/2105.13922/) (2021)
- [Toward Equation of Motion for Deep Neural Networks: Continuous-time Gradient Descent and Discretization Error Analysis](https://mlsanity.com/paper/2210.15898/) (2022)
- [On a continuous time model of gradient descent dynamics and instability in deep learning](https://mlsanity.com/paper/2302.01952/) (2023)
- [Fast Diffusion Probabilistic Model Sampling through the lens of Backward Error Analysis](https://mlsanity.com/paper/2304.11446/) (2023)
- [The Geometric Occam's Razor Implicit in Deep Learning](https://mlsanity.com/paper/2111.15090/) (2021)
- [Create and Find Flatness: Building Flat Training Spaces in Advance for Continual Learning](https://mlsanity.com/paper/2309.11305/) (2023)
- [Implicit regularisation in stochastic gradient descent: from single-objective to two-player games](https://mlsanity.com/paper/2307.05789/) (2023)

## Built on by (most cited first, 4 of 4)

- [Commute Your Domains: Trajectory Optimality Criterion for Multi-Domain Learning](https://mlsanity.com/paper/2501.15556/) (2025)
- [Optimizer Memory Makes Shuffle Order a First-Order Source of Fine-Tuning Noise](https://mlsanity.com/paper/2606.29554/) (2026)
- [Majority Kernels: An Approach to Leverage Big Model Dynamics for Efficient Small Model Training](https://mlsanity.com/paper/2402.05033/) (2024)
- [Intrinsic-Extrinsic Coupling in Learning Dynamics](https://mlsanity.com/paper/2609.30185/) (2026)

## Abstract (arXiv)

Using backward error analysis, we compute implicit training biases in multitask and continual learning settings for neural networks trained with stochastic gradient descent. In particular, we derive modified losses that are implicitly minimized during training. They have three terms: the original loss, accounting for convergence, an implicit flatness regularization term proportional to the learning rate, and a last term, the conflict term, which can theoretically be detrimental to both convergence and implicit regularization. In multitask, the conflict term is a well-known quantity, measuring the gradient alignment between the tasks, while in continual learning the conflict term is a new quantity in deep learning optimization, although a basic tool in differential geometry: The Lie bracket between the task gradients.