mlsanityData to 7 Oct 2026

Learning-to-Learn Stochastic Gradient Descent with Biased Regularization

arXiv 1903.10399 ↗First posted 25 Mar 2019cs.LG stat.MLcited by 49

Not on the map. Why?

Abstract (arXiv)

We study the problem of learning-to-learn: inferring a learning algorithm that works well on tasks sampled from an unknown distribution. As class of algorithms we consider Stochastic Gradient Descent on the true risk regularized by the square euclidean distance to a bias vector. We present an average excess risk bound for such a learning algorithm. This result quantifies the potential benefit of using a bias vector with respect to the unbiased case. We then address the problem of estimating the bias from a sequence of tasks. We propose a meta-algorithm which incrementally updates the bias, as new tasks are observed. The low space and time complexity of this approach makes it appealing in practice. We provide guarantees on the learning ability of the meta-algorithm. A key feature of our results is that, when the number of tasks grows and their variance is relatively small, our learning-to-learn approach has a significant advantage over learning each task in isolation by Stochastic Gradient Descent without a bias term. We report on numerical experiments which demonstrate the effectiveness of our approach.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Mar 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 14; 2022: 14; 2023: 7; 2024: 8; 2025: 3; 2026: 2.

49 papers.

Sort:

By area:

  1. 2021Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyondcited by 151
  2. 2021Generalization Bounds For Meta-Learning: An Information-Theoretic Analysiscited by 45
  3. 2020Provable Representation Learning for Imitation Learning via Bi-level Optimizationcited by 41
  4. 2022Adaptive and Robust Multi-Task Learningcited by 33
  5. 2024A Statistical Theory of Regularization-Based Continual Learningcited by 32
  6. 2021On the Power of Multitask Representation Learning in Linear MDPcited by 30
  7. 2021Meta-Adaptive Nonlinear Control: Theory and Algorithmscited by 23
  8. 2021On Data Efficiency of Meta-learningcited by 16
  9. 2022Nearly Minimax Algorithms for Linear Bandits with Shared Representationcited by 16
  10. 2024Imperative Learning: A Self-supervised Neuro-Symbolic Learning Framework for Robot Autonomycited by 16
  11. 2021Meta-Learning with Neural Tangent Kernelscited by 15
  12. 2021Learning-to-learn non-convex piecewise-Lipschitz functionscited by 15
37 more
  1. 2023Towards Constituting Mathematical Structures for Learning to Optimizecited by 13
  2. 2021Conditional Meta-Learning of Linear Representationscited by 12
  3. 2024A CMDP-within-online framework for Meta-Safe Reinforcement Learningcited by 12
  4. 2021An Information-Theoretic Analysis of the Impact of Task Similarity on Meta-Learningcited by 10
  5. 2022Trace norm regularization for multi-task learning with scarce datacited by 10
  6. 2024More Flexible PAC-Bayesian Meta-Learning by Learning Learning Algorithmscited by 10
  7. 2021Learning an Explicit Hyperparameter Prediction Function Conditioned on Taskscited by 9
  8. 2022Provable Generalization of Overparameterized Meta-learning Trained with SGDcited by 9
  9. 2023A Simple Yet Effective Strategy to Robustify the Meta Learning Paradigmcited by 9
  10. 2022Non-stationary Bandits and Meta-Learning with a Small Set of Optimal Armscited by 8
  11. 2023Bayes meets Bernstein at the Meta Level: an Analysis of Fast Rates in Meta-Learning with PAC-Bayescited by 8
  12. 2023Nonlinear Meta-Learning Can Guarantee Faster Ratescited by 8
  13. 2021Meta-Learning with Graph Neural Networks: Methods and Applicationscited by 7
  14. 2021Meta-strategy for Learning Tuning Parameters with Guaranteescited by 6
  15. 2022Meta Representation Learning with Contextual Linear Banditscited by 6
  16. 2024Reawakening knowledge: Anticipatory recovery from catastrophic interference via structured trainingcited by 6
  17. 2022Meta-Learning Adversarial Banditscited by 5
  18. 2021Adaptive Clustering and Personalization in Multi-Agent Stochastic Linear Banditscited by 4
  19. 2022Meta Learning MDPs with Linear Transition Modelscited by 4
  20. 2022Multi-Environment Meta-Learning in Stochastic Linear Banditscited by 4
  21. 2023DAC-MR: Data Augmentation Consistency Based Meta-Regularization for Meta-Learningcited by 4
  22. 2023Hypothesis Transfer Learning with Surrogate Classification Losses: Generalization Bounds through Algorithmic Stabilitycited by 4
  23. 2021Dynamic Regret Analysis for Online Meta-Learningcited by 3
  24. 2022A General framework for PAC-Bayes Bounds for Meta-Learningcited by 3
  25. 2024Leveraging Gradients for Unsupervised Accuracy Estimation under Distribution Shiftcited by 3
  26. 2024Theoretical Investigations and Practical Enhancements on Tail Task Risk Minimization in Meta Learningcited by 3
  27. 2022On Hypothesis Transfer Learning of Functional Linear Modelscited by 2
  28. 2022Improving Meta-Learning Generalization with Activation-Based Early-Stoppingcited by 2
  29. 2024Meta-Reinforcement Learning with Universal Policy Adaptation: Provable Near-Optimality under All-task Optimum Comparatorcited by 2
  30. 2022Learning Tensor Representations for Meta-Learningcited by 1
  31. 2022EEML: Ensemble Embedded Meta-learningcited by 1
  32. 2025Gradient Descent with Provably Tuned Learning-rate Schedulescited by 1
  33. 2026When More Data Doesn't Help: Limits of Adaptation in Multitask Learningcited by 1
  34. 2023Learning to Relax: Setting Solver Parameters Across a Sequence of Linear System Instancescited by 0
  35. 2025Meta-learning of shared linear representations beyond well-specified linear regressioncited by 0
  36. 2025Neural Coherence : Find higher performance to out-of-distribution tasks from few samplescited by 0
  37. 2026The Geometry of Saturation: Effective Rank Predicts When Labels Stop Helping in Few-Shot Classificationcited by 0