Data to 7 Oct 2026

Meta-Learning with Implicit Gradients

arXiv 1909.04630 ↗First posted 10 Sep 2019cs.LG cs.AI math.OCcited by 523

Not on the map. Why?

Abstract (arXiv)

A core capability of intelligent systems is the ability to quickly learn new tasks by drawing on prior experience. Gradient (or optimization) based meta-learning has recently emerged as an effective approach for few-shot learning. In this formulation, meta-parameters are learned in the outer loop, while task-specific models are learned in the inner-loop, by using only a small amount of data from the current task. A key challenge in scaling these approaches is the need to differentiate through the inner loop learning process, which can impose considerable computational and memory burdens. By drawing upon implicit differentiation, we develop the implicit MAML algorithm, which depends only on the solution to the inner level optimization and not the path taken by the inner loop optimizer. This effectively decouples the meta-gradient computation from the choice of inner loop optimizer. As a result, our approach is agnostic to the choice of inner loop optimizer and can gracefully handle many gradient steps without vanishing gradients or memory constraints. Theoretically, we prove that implicit MAML can compute accurate meta-gradients with a memory footprint that is, up to small constant factors, no more than that which is required to compute a single inner loop gradient and at no overall increase in the total computational cost. Experimentally, we show that these benefits of implicit MAML translate into empirical gains on few-shot image recognition benchmarks.

Key facts. Cited by 523 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 2; 2021: 109; 2022: 107; 2023: 104; 2024: 99; 2025: 58; 2026: 44.

523 papers.

Sort:

By area:

2021True Few-Shot Learning with Language Modelscited by 343
2021Efficient and Modular Implicit Differentiationcited by 199
2022Dataset Distillation using Neural Feature Regressioncited by 177
2021Meta-learning via Language Model In-context Tuningcited by 163
2022A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunitiescited by 162
2021Adversarial Unlearning of Backdoors via Implicit Hypergradientcited by 160
2021Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyondcited by 151
2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 143
2021A Near-Optimal Algorithm for Stochastic Bilevel Optimization via Double-Momentumcited by 106
2021Provably Faster Algorithms for Bilevel Optimizationcited by 106
2022BOME! Bilevel Optimization Made Easy: A Simple First-Order Approachcited by 103
2023A Fully First-Order Method for Stochastic Bilevel Optimizationcited by 97

38 more

2020Offline Meta-Reinforcement Learning with Advantage Weightingcited by 92
2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 89
2023On Penalty-based Bilevel Gradient Descent Methodcited by 87
2021Multi-Objective Loss Balancing for Physics-Informed Deep Learningcited by 86
2021JFB: Jacobian-Free Backpropagation for Implicit Networkscited by 84
2022Tutorial on amortized optimizationcited by 83
2021Gradients are Not All You Needcited by 82
2022COIN++: Neural Compression Across Modalitiescited by 81
2022Transformers as Meta-Learners for Implicit Neural Representationscited by 79
2022When Do Flat Minima Optimizers Work?cited by 78
2021Is Attention Better Than Matrix Decomposition?cited by 75
2021Revisiting and Advancing Fast Adversarial Training Through The Lens of Bi-Level Optimizationcited by 75
2021On Training Implicit Modelscited by 73
2021A Single-Timescale Method for Stochastic Bilevel Optimizationcited by 69
2023Advances and Challenges in Meta-Learning: A Technical Reviewcited by 65
2021Towards Gradient-based Bilevel Optimization with Non-convex Followers and Beyondcited by 62
2021Amortized Implicit Differentiation for Stochastic Bilevel Optimizationcited by 62
2021A Value-Function-based Interior-point Method for Non-convex Bi-level Optimizationcited by 60
2022Improving Object-centric Learning with Query Optimizationcited by 60
2022Sharp-MAML: Sharpness-Aware Model-Agnostic Meta Learningcited by 57
2022Advancing Model Pruning via Bi-level Optimizationcited by 55
2022Meta Balanced Network for Fair Face Recognitioncited by 53
2024Erasing Undesirable Influence in Diffusion Modelscited by 52
2021iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradientscited by 51
2021Meta-learning PINN loss functionscited by 51
2021Rectifying the Shortcut Learning of Background for Few-Shot Learningcited by 51
2021Bridging Multi-Task Learning and Meta-Learning: Towards Efficient Training and Effective Adaptationcited by 50
2023On Penalty Methods for Nonconvex Bilevel Optimization and First-Order Stochastic Approximationcited by 50
2021Meta-Learning with Task-Adaptive Loss Function for Few-Shot Learningcited by 49
2022Object Representations as Fixed Points: Training Iterative Refinement Algorithms with Implicit Differentiationcited by 46
2023A Closer Look at Few-shot Classification Againcited by 46
2023Dataset Distillation with Convexified Implicit Gradientscited by 46
2021Meta-Learning with Fewer Tasks through Task Interpolationcited by 45
2020Provable Representation Learning for Imitation Learning via Bi-level Optimizationcited by 41
2021How Fine-Tuning Allows for Effective Meta-Learningcited by 41
2023CLIP-guided Prototype Modulating for Few-shot Action Recognitioncited by 41
2024Principled Penalty-based Methods for Bilevel Reinforcement Learning and RLHFcited by 41

50 most cited of 523 All 523 (JSON)