mlsanityData to 7 Oct 2026

Global Convergence of MAML and Theory-Inspired Neural Architecture Search for Few-Shot Learning

arXiv 2203.09137 ↗First posted 17 Mar 2022cs.LG cs.CV stat.MLcited by 22

Not on the map. Why?

Abstract (arXiv)

Model-agnostic meta-learning (MAML) and its variants have become popular approaches for few-shot learning. However, due to the non-convexity of deep neural nets (DNNs) and the bi-level formulation of MAML, the theoretical properties of MAML with DNNs remain largely unknown. In this paper, we first prove that MAML with over-parameterized DNNs is guaranteed to converge to global optima at a linear rate. Our convergence analysis indicates that MAML with over-parameterized DNNs is equivalent to kernel regression with a novel class of kernels, which we name as Meta Neural Tangent Kernels (MetaNTK). Then, we propose MetaNTK-NAS, a new training-free neural architecture search (NAS) method for few-shot learning that uses MetaNTK to rank and select architectures. Empirically, we compare our MetaNTK-NAS with previous NAS methods on two popular few-shot learning benchmarks, miniImageNet, and tieredImageNet. We show that the performance of MetaNTK-NAS is comparable or better than the state-of-the-art NAS method designed for few-shot learning while enjoying more than 100x speedup. We believe the efficiency of MetaNTK-NAS makes itself more practical for many real-world tasks.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 56 of its 79 references that are in the map. First posted on arXiv 17 Mar 2022.

Builds on

56 of 79 references.

References · 56

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  6. 2016Wide Residual Networkscited by 3,057
  7. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  8. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  9. 2016Matching Networks for One Shot Learningcited by 2,246
  10. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  11. 2018Group Normalizationcited by 1,514
  12. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
38 more
  1. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  2. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  3. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  4. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  5. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  6. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  7. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  8. 2019Meta-Learning with Implicit Gradientscited by 523
  9. 2019Meta-Learning with Differentiable Convex Optimizationcited by 461
  10. 2018Meta-Learning for Semi-Supervised Few-Shot Classificationcited by 456
  11. 2020Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?cited by 435
  12. 2018Meta-learning with differentiable closed-form solverscited by 395
  13. 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 375
  14. 2019Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAMLcited by 359
  15. 2018How to train your MAMLcited by 306
  16. 2017Meta Networkscited by 268
  17. 2018Probabilistic Model-Agnostic Meta-Learningcited by 254
  18. 2018Meta-Learning: A Surveycited by 254
  19. 2020Neural Architecture Search without Trainingcited by 241
  20. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 221
  21. 2016Learning to Optimizecited by 195
  22. 2021Zero-Cost Proxies for Lightweight NAScited by 194
  23. 2019Adaptive Gradient-Based Meta-Learning Methodscited by 183
  24. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  25. 2021Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspectivecited by 168
  26. 2019Online Meta-Learningcited by 137
  27. 2019On the Convergence Theory of Gradient-Based Model-Agnostic Meta-Learning Algorithmscited by 132
  28. 2019Evaluating the Search Phase of Neural Architecture Searchcited by 120
  29. 2021Rethinking Architecture Selection in Differentiable NAScited by 92
  30. 2019Meta-Learning of Neural Architectures for Few-Shot Learningcited by 73
  31. 2019Provable Guarantees for Gradient-Based Meta-Learningcited by 67
  32. 2018Diverse Few-Shot Text Classification with Multiple Metricscited by 61
  33. 2019Learning to Few-Shot Learn Across Diverse Natural Language Classification Taskscited by 57
  34. 2021A Survey on Green Deep Learningcited by 52
  35. 2021Bridging Multi-Task Learning and Meta-Learning: Towards Efficient Training and Effective Adaptationcited by 50
  36. 2021KNAS: Green Neural Architecture Searchcited by 33
  37. 2020How Important is the Train-Validation Split in Meta-Learning?cited by 30
  38. 2020Theoretical Convergence of Multi-Step Model-Agnostic Meta-Learningcited by 28

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 9; 2024: 6; 2025: 2; 2026: 2.

22 papers.

Sort:

By area:

  1. 2023NTK-SAP: Improving neural network pruning by aligning training dynamicscited by 30
  2. 2023LEMON: Lossless model expansioncited by 23
  3. 2024Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Surveycited by 23
  4. 2024Neural Lineagecited by 9
  5. 2023Balanced Training for Sparse GANscited by 8
  6. 2023GenCo: An Auxiliary Generator from Contrastive Learning for Enhanced Few-Shot Learning in Remote Sensingcited by 8
  7. 2024NTK-Guided Few-Shot Class Incremental Learningcited by 7
  8. 2023Convergence of Gradient-based MAML in LQRcited by 5
  9. 2022Meta-ticket: Finding optimal subnetworks for few-shot learning within randomly initialized neural networkscited by 4
  10. 2022Revisiting Few-Shot Learning from a Causal Perspectivecited by 4
  11. 2022Future Gradient Descent for Adapting the Temporal Shifting Data Distribution in Online Recommendation Systemscited by 3
  12. 2023Disentangled Representation Learning with Transmitted Information Bottleneckcited by 3
10 more
  1. 2023Optimistic Meta-Gradientscited by 2
  2. 2023STDA-Meta: A Meta-Learning Framework for Few-Shot Traffic Predictioncited by 1
  3. 2023Invariant-Feature Subspace Recovery: A New Class of Provable Domain Generalization Algorithmscited by 1
  4. 2024Task Attribute Distance for Few-Shot Learning: Theoretical Analysis and Applicationscited by 1
  5. 2024Meta-Task: A Method-Agnostic Framework for Learning to Regularize in Few-Shot Learningcited by 0
  6. 2024Local Descriptors Weighted Adaptive Threshold Filtering For Few-Shot Learningcited by 0
  7. 2025A2M2-Net: Adaptively Aligned Multi-Scale Moment for Few-Shot Action Recognitioncited by 0
  8. 2025Directed-MAML: Meta Reinforcement Learning Algorithm with Task-directed Approximationcited by 0
  9. 2026Binomial Gradient-Based Meta-Learning for Enhanced Meta-Gradient Estimationcited by 0
  10. 2026From Regression to Inference: Meta-Learning Predictors for Neural Architecture Searchcited by 0