mlsanityData to 14 Oct 2026

An Analytical Theory of Auxiliary Learning

arXiv 2609.29774 ↗First posted 24 Sep 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Auxiliary learning is an optimization paradigm in which a neural network's performance on a target task is improved by jointly training it on additional tasks. However, the mechanisms behind this improvement remain poorly understood. We study this problem using a teacher-student framework and derive a closed system of differential equations describing the dynamics of online stochastic gradient descent in the large-input limit. For linear networks, we obtain a closed-form expression for the generalization error to leading order in the learning rate, quantifying how task correlations and label noise determine the benefit of auxiliary learning. For non-linear activation functions, we develop a fluctuation-dissipation analytical theory that establishes a general relation linking the main and auxiliary errors to the corresponding single-task error. Numerical experiments support the theoretical predictions and show how auxiliary tasks improve generalization by balancing the forcing dynamics towards the optimal solution with gradient noise.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 14 of its 32 references that are in the map. First posted on arXiv 24 Sep 2026.

Builds on

14 of 32 references.

References · 14

  1. 2017A Survey on Multi-Task Learningcited by 840
  2. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 803
  3. 2020Multi-Task Learning with Deep Neural Networks: A Surveycited by 375
  4. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 300
  5. 2017Three Factors Influencing Minima in SGDcited by 273
  6. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  7. 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
  8. 2018Auxiliary Tasks in Multi-task Learningcited by 108
  9. 2021Continual Learning in the Teacher-Student Setup: Impact of Task Similaritycited by 88
  10. 2018Fluctuation-dissipation relations for stochastic gradient descentcited by 48
  11. 2021An Analytical Theory of Curriculum Learning in Teacher-Student Networkscited by 27
  12. 2021Statistical Mechanical Analysis of Catastrophic Forgetting in Continual Learning with Teacher and Student Networkscited by 22
2 more
  1. 2019On-line learning dynamics of ReLU neural networks using statistical physics techniquescited by 4
  2. 2025Analytic theory of dropout regularizationcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025A Teacher-Student Perspective on the Dynamics of Learning Near the Optimal Point
  2. 2025Generating Auxiliary Tasks with Reinforcement Learning
  3. 2026Dichotomy of Feature Learning and Unlearning: Fast-Slow Analysis on Neural Networks with Stochastic Gradient Descent
  4. 2024Dynamics of Supervised and Reinforcement Learning in the Non-Linear Perceptron
  5. 2024Ornstein-Uhlenbeck Adaptation as a Mechanism for Learning in Brains and Machines
  6. 2025High-Dimensional Learning Dynamics of Quantized Models with Straight-Through Estimator
  7. 2026A Boundary-Layer Mechanism for One-Third Scaling in Online Softmax Classification
  8. 2026Is Backpropagation Optimal? When Synthetic Gradients Improve Sample Efficiency