# Global Convergence of MAML and Theory-Inspired Neural Architecture Search for Few-Shot Learning

arXiv: https://arxiv.org/abs/2203.09137 · HTML: https://mlsanity.com/paper/2203.09137/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 56 of its 79 references that are in the map.
- First posted on arXiv 17 Mar 2022.

## Builds on (most cited first, 30 of 56)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [Meta-Learning in Neural Networks: A Survey](https://mlsanity.com/paper/2004.05439/) (2020)
- [A Convergence Theory for Deep Learning via Over-Parameterization](https://mlsanity.com/paper/1811.03962/) (2018)
- [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019)
- [Gradient Descent Finds Global Minima of Deep Neural Networks](https://mlsanity.com/paper/1811.03804/) (2018)
- [On Exact Computation with an Infinitely Wide Neural Net](https://mlsanity.com/paper/1904.11955/) (2019)
- [Meta-Learning with Implicit Gradients](https://mlsanity.com/paper/1909.04630/) (2019)
- [Meta-Learning with Differentiable Convex Optimization](https://mlsanity.com/paper/1904.03758/) (2019)
- [Meta-Learning for Semi-Supervised Few-Shot Classification](https://mlsanity.com/paper/1803.00676/) (2018)
- [Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?](https://mlsanity.com/paper/2003.11539/) (2020)
- [Meta-learning with differentiable closed-form solvers](https://mlsanity.com/paper/1805.08136/) (2018)
- [Opacus: User-Friendly Differential Privacy Library in PyTorch](https://mlsanity.com/paper/2109.12298/) (2021)
- [Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML](https://mlsanity.com/paper/1909.09157/) (2019)
- [How to train your MAML](https://mlsanity.com/paper/1810.09502/) (2018)
- [Meta Networks](https://mlsanity.com/paper/1703.00837/) (2017)
- [Probabilistic Model-Agnostic Meta-Learning](https://mlsanity.com/paper/1806.02817/) (2018)
- [Meta-Learning: A Survey](https://mlsanity.com/paper/1810.03548/) (2018)

## Built on by (most cited first, 22 of 22)

- [NTK-SAP: Improving neural network pruning by aligning training dynamics](https://mlsanity.com/paper/2304.02840/) (2023)
- [LEMON: Lossless model expansion](https://mlsanity.com/paper/2310.07999/) (2023)
- [Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Survey](https://mlsanity.com/paper/2406.05392/) (2024)
- [Neural Lineage](https://mlsanity.com/paper/2406.11129/) (2024)
- [Balanced Training for Sparse GANs](https://mlsanity.com/paper/2302.14670/) (2023)
- [GenCo: An Auxiliary Generator from Contrastive Learning for Enhanced Few-Shot Learning in Remote Sensing](https://mlsanity.com/paper/2307.14612/) (2023)
- [NTK-Guided Few-Shot Class Incremental Learning](https://mlsanity.com/paper/2403.12486/) (2024)
- [Convergence of Gradient-based MAML in LQR](https://mlsanity.com/paper/2309.06588/) (2023)
- [Meta-ticket: Finding optimal subnetworks for few-shot learning within randomly initialized neural networks](https://mlsanity.com/paper/2205.15619/) (2022)
- [Revisiting Few-Shot Learning from a Causal Perspective](https://mlsanity.com/paper/2209.13816/) (2022)
- [Future Gradient Descent for Adapting the Temporal Shifting Data Distribution in Online Recommendation Systems](https://mlsanity.com/paper/2209.01143/) (2022)
- [Disentangled Representation Learning with Transmitted Information Bottleneck](https://mlsanity.com/paper/2311.01686/) (2023)
- [Optimistic Meta-Gradients](https://mlsanity.com/paper/2301.03236/) (2023)
- [STDA-Meta: A Meta-Learning Framework for Few-Shot Traffic Prediction](https://mlsanity.com/paper/2310.20223/) (2023)
- [Invariant-Feature Subspace Recovery: A New Class of Provable Domain Generalization Algorithms](https://mlsanity.com/paper/2311.00966/) (2023)
- [Task Attribute Distance for Few-Shot Learning: Theoretical Analysis and Applications](https://mlsanity.com/paper/2403.03535/) (2024)
- [Meta-Task: A Method-Agnostic Framework for Learning to Regularize in Few-Shot Learning](https://mlsanity.com/paper/2402.18599/) (2024)
- [Local Descriptors Weighted Adaptive Threshold Filtering For Few-Shot Learning](https://mlsanity.com/paper/2408.15924/) (2024)
- [A2M2-Net: Adaptively Aligned Multi-Scale Moment for Few-Shot Action Recognition](https://mlsanity.com/paper/2509.17638/) (2025)
- [Directed-MAML: Meta Reinforcement Learning Algorithm with Task-directed Approximation](https://mlsanity.com/paper/2510.00212/) (2025)
- [Binomial Gradient-Based Meta-Learning for Enhanced Meta-Gradient Estimation](https://mlsanity.com/paper/2604.13263/) (2026)
- [From Regression to Inference: Meta-Learning Predictors for Neural Architecture Search](https://mlsanity.com/paper/2605.09290/) (2026)

## Abstract (arXiv)

Model-agnostic meta-learning (MAML) and its variants have become popular approaches for few-shot learning. However, due to the non-convexity of deep neural nets (DNNs) and the bi-level formulation of MAML, the theoretical properties of MAML with DNNs remain largely unknown. In this paper, we first prove that MAML with over-parameterized DNNs is guaranteed to converge to global optima at a linear rate. Our convergence analysis indicates that MAML with over-parameterized DNNs is equivalent to kernel regression with a novel class of kernels, which we name as Meta Neural Tangent Kernels (MetaNTK). Then, we propose MetaNTK-NAS, a new training-free neural architecture search (NAS) method for few-shot learning that uses MetaNTK to rank and select architectures. Empirically, we compare our MetaNTK-NAS with previous NAS methods on two popular few-shot learning benchmarks, miniImageNet, and tieredImageNet. We show that the performance of MetaNTK-NAS is comparable or better than the state-of-the-art NAS method designed for few-shot learning while enjoying more than 100x speedup. We believe the efficiency of MetaNTK-NAS makes itself more practical for many real-world tasks.