mlsanityData to 14 Oct 2026

Memory-Based Optimization Methods for Model-Agnostic Meta-Learning and Personalized Federated Learning

arXiv 2106.04911 ↗First posted 9 Jun 2021cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

In recent years, model-agnostic meta-learning (MAML) has become a popular research area. However, the stochastic optimization of MAML is still underdeveloped. Existing MAML algorithms rely on the ``episode'' idea by sampling a few tasks and data points to update the meta-model at each iteration. Nonetheless, these algorithms either fail to guarantee convergence with a constant mini-batch size or require processing a large number of tasks at every iteration, which is unsuitable for continual learning or cross-device federated learning where only a small number of tasks are available per iteration or per round. To address these issues, this paper proposes memory-based stochastic algorithms for MAML that converge with vanishing error. The proposed algorithms require sampling a constant number of tasks and data samples per iteration, making them suitable for the continual learning scenario. Moreover, we introduce a communication-efficient memory-based MAML algorithm for personalized federated learning in cross-device (with client sampling) and cross-silo (without client sampling) settings. Our theoretical analysis improves the optimization theory for MAML, and our empirical results corroborate our theoretical findings. Interested readers can access our code at urlhttps://github.com/bokun-wang/moml.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 43 of its 59 references that are in the map. First posted on arXiv 9 Jun 2021.

Builds on

43 of 59 references.

References · 43

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  4. 2019Advances and Open Problems in Federated Learningcited by 3,046
  5. 2017Prototypical Networks for Few-shot Learningcited by 3,015
  6. 2016Matching Networks for One Shot Learningcited by 2,255
  7. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  8. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  9. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  10. 2018On First-Order Meta-Learning Algorithmscited by 1,018
  11. 2020Meta-Learning in Neural Networks: A Surveycited by 950
  12. 2020Personalized Federated Learning with Moreau Envelopescited by 618
31 more
  1. 2018Local SGD Converges Fast and Communicates Littlecited by 605
  2. 2019Meta-Learning with Implicit Gradientscited by 525
  3. 2018Bilevel Programming for Hyperparameter Optimization and Meta-Learningcited by 405
  4. 2017Meta-SGD: Learning to Learn Quickly for Few-Shot Learningcited by 379
  5. 2019Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAMLcited by 359
  6. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 312
  7. 2018How to train your MAMLcited by 309
  8. 2019Improving Federated Learning Personalization via Model Agnostic Meta Learningcited by 269
  9. 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
  10. 2018Probabilistic Model-Agnostic Meta-Learningcited by 256
  11. 2018Deep Learning: A Critical Appraisalcited by 217
  12. 2020Federated Learning of a Mixture of Global and Local Modelscited by 210
  13. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 202
  14. 2018Bayesian Model-Agnostic Meta-Learningcited by 194
  15. 2018Recasting Gradient-Based Meta-Learning as Hierarchical Bayescited by 180
  16. 2019SCAFFOLD: Stochastic Controlled Averaging for Federated Learningcited by 175
  17. 2020Is Local SGD Better than Minibatch SGD?cited by 151
  18. 2019On the Convergence Theory of Gradient-Based Model-Agnostic Meta-Learning Algorithmscited by 132
  19. 2021Distributionally Robust Federated Averagingcited by 104
  20. 2014Stochastic Compositional Gradient Descent: Algorithms for Minimizing Compositions of Expected-Value Functionscited by 95
  21. 2019Provable Guarantees for Gradient-Based Meta-Learningcited by 67
  22. 2020Convergence of Meta-Learning with Task-Specific Adaptation over Partial Parameterscited by 65
  23. 2019ES-MAML: Simple Hessian-Free Meta Learningcited by 57
  24. 2018A Single Time-Scale Stochastic Approximation Method for Nested Stochastic Optimizationcited by 53
  25. 2020Solving Stochastic Compositional Optimization is Nearly as Easy as Solving Stochastic Optimizationcited by 51
  26. 2020Biased Stochastic First-Order Methods for Conditional Stochastic Optimization and Applications in Meta Learningcited by 47
  27. 2020Theoretical Convergence of Multi-Step Model-Agnostic Meta-Learningcited by 28
  28. 2019Alpha MAML: Adaptive Model-Agnostic Meta-Learningcited by 25
  29. 2020On the Global Optimality of Model-Agnostic Meta-Learningcited by 21
  30. 2020Tailoring: encoding inductive biases by optimizing unsupervised objectives at prediction timecited by 13
  31. 2020Compositional ADAM: An Adaptive Compositional Solvercited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 3; 2025: 0; 2026: 1.

6 papers.

  1. 2023Contextual Stochastic Bilevel Optimizationcited by 14
  2. 2024SMILE: Speech Meta In-Context Learning for Low-Resource Language Automatic Speech Recognitioncited by 4
  3. 2024Pre-Training and Personalized Fine-Tuning via Over-the-Air Federated Meta-Learning: Convergence-Generalization Trade-Offscited by 3
  4. 2024Agnostic Sharpness-Aware Minimizationcited by 2
  5. 2023Achieving Linear Speedup in Decentralized Stochastic Compositional Minimax Optimizationcited by 0
  6. 2026Federated Compositional Muon Optimizer for Matrix-Wise Modelscited by 0