mlsanityData to 7 Oct 2026

Meta-Learning with Neural Tangent Kernels

arXiv 2102.03909 ↗First posted 7 Feb 2021cs.LG cs.AIcited by 15

Not on the map. Why?

Abstract (arXiv)

Model Agnostic Meta-Learning (MAML) has emerged as a standard framework for meta-learning, where a meta-model is learned with the ability of fast adapting to new tasks. However, as a double-looped optimization problem, MAML needs to differentiate through the whole inner-loop optimization path for every outer-loop training step, which may lead to both computational inefficiency and sub-optimal solutions. In this paper, we generalize MAML to allow meta-learning to be defined in function spaces, and propose the first meta-learning paradigm in the Reproducing Kernel Hilbert Space (RKHS) induced by the meta-model's Neural Tangent Kernel (NTK). Within this paradigm, we introduce two meta-learning algorithms in the RKHS, which no longer need a sub-optimal iterative inner-loop adaptation as in the MAML framework. We achieve this goal by 1) replacing the adaptation with a fast-adaptive regularizer in the RKHS; and 2) solving the adaptation analytically based on the NTK theory. Extensive experimental studies demonstrate advantages of our paradigm in both efficiency and quality of solutions compared to related meta-learning algorithms. Another interesting feature of our proposed methods is that they are demonstrated to be more robust to adversarial attacks and out-of-distribution adaptation than popular baselines, as demonstrated in our experiments.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 31 of its 41 references that are in the map. First posted on arXiv 7 Feb 2021.

Builds on

31 of 41 references.

References · 31

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  6. 2016Matching Networks for One Shot Learningcited by 2,246
  7. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  8. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  9. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  10. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  11. 2014Neural Turing Machinescited by 630
  12. 2016Learning to learn by gradient descent by gradient descentcited by 616
19 more
  1. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  2. 2019Meta-Learning with Implicit Gradientscited by 523
  3. 2017Countering Adversarial Images using Input Transformationscited by 517
  4. 2018TADAM: Task dependent adaptive metric for improved few-shot learningcited by 408
  5. 2019Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examplescited by 312
  6. 2018Probabilistic Model-Agnostic Meta-Learningcited by 254
  7. 2019Simple Black-box Adversarial Attackscited by 249
  8. 2018Adversarial Risk and the Dangers of Evaluating Against Weak Attackscited by 221
  9. 2018Bayesian Model-Agnostic Meta-Learningcited by 194
  10. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 185
  11. 2019Adaptive Gradient-Based Meta-Learning Methodscited by 183
  12. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 164
  13. 2020Provable Meta-Learning of Linear Representationscited by 161
  14. 2019On the Convergence Theory of Gradient-Based Model-Agnostic Meta-Learning Algorithmscited by 132
  15. 2019Provable Guarantees for Gradient-Based Meta-Learningcited by 67
  16. 2019Hierarchically Structured Meta-learningcited by 67
  17. 2019Learning-to-Learn Stochastic Gradient Descent with Biased Regularizationcited by 49
  18. 2019Learning to Balance: Bayesian Meta-Learning for Imbalanced and Out-of-distribution Taskscited by 48
  19. 2019Meta-learnt priors slow down catastrophic forgetting in neural networkscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 7; 2023: 3; 2024: 1; 2025: 0; 2026: 1.

15 papers.

Sort:

  1. 2022Fast Finite Width Neural Tangent Kernelcited by 62
  2. 2021Meta-Learning with Task-Adaptive Loss Function for Few-Shot Learningcited by 49
  3. 2022A Fast, Well-Founded Approximation to the Empirical Neural Tangent Kernelcited by 30
  4. 2022Making Look-Ahead Active Learning Strategies Feasible with Neural Tangent Kernelscited by 25
  5. 2022Fast Neural Kernel Embeddings for General Activationscited by 18
  6. 2023Learning to Learn from APIs: Black-Box Data-Free Meta-Learningcited by 18
  7. 2021Learning-to-learn non-convex piecewise-Lipschitz functionscited by 15
  8. 2022Few-shot Backdoor Attacks via Neural Tangent Kernelscited by 15
  9. 2022Meta-Learning with Less Forgetting on Large-Scale Non-Stationary Task Distributionscited by 14
  10. 2023Understanding Reconstruction Attacks with the Neural Tangent Kernel and Dataset Distillationcited by 14
  11. 2022Continuous-Time Meta-Learning with Forward Mode Differentiationcited by 12
  12. 2021Meta Learning on a Sequence of Imbalanced Domains with Difficulty Awarenesscited by 8
3 more
  1. 2024lpNTK: Better Generalisation with Less Data via Sample Interaction During Learningcited by 6
  2. 2023SemiGPC: Distribution-Aware Label Refinement for Imbalanced Semi-Supervised Learning Using Gaussian Processescited by 0
  3. 2026Kernel Reboot: Breaking the Boundaries of Neural Tangent Kernels for Neural Fieldscited by 0