# Meta-Learning with Adjoint Methods

arXiv: https://arxiv.org/abs/2110.08432 · HTML: https://mlsanity.com/paper/2110.08432/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 63 references that are in the map.
- First posted on arXiv 16 Oct 2021.

## Builds on (most cited first, 30 of 35)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [Meta-Learning in Neural Networks: A Survey](https://mlsanity.com/paper/2004.05439/) (2020)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [Meta-Learning with Implicit Gradients](https://mlsanity.com/paper/1909.04630/) (2019)
- [Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm](https://mlsanity.com/paper/1608.04471/) (2016)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Meta-Learning with Latent Embedding Optimization](https://mlsanity.com/paper/1807.05960/) (2018)
- [Meta-Learning with Differentiable Convex Optimization](https://mlsanity.com/paper/1904.03758/) (2019)
- [TADAM: Task dependent adaptive metric for improved few-shot learning](https://mlsanity.com/paper/1805.10123/) (2018)
- [Meta-learning with differentiable closed-form solvers](https://mlsanity.com/paper/1805.08136/) (2018)
- [Meta-SGD: Learning to Learn Quickly for Few-Shot Learning](https://mlsanity.com/paper/1707.09835/) (2017)
- [Learning to reinforcement learn](https://mlsanity.com/paper/1611.05763/) (2016)
- [Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples](https://mlsanity.com/paper/1903.03096/) (2019)
- [Meta Networks](https://mlsanity.com/paper/1703.00837/) (2017)
- [Probabilistic Model-Agnostic Meta-Learning](https://mlsanity.com/paper/1806.02817/) (2018)
- [Learning to Optimize](https://mlsanity.com/paper/1606.01885/) (2016)
- [Bayesian Model-Agnostic Meta-Learning](https://mlsanity.com/paper/1806.03836/) (2018)
- [Recasting Gradient-Based Meta-Learning as Hierarchical Bayes](https://mlsanity.com/paper/1801.08930/) (2018)
- [Infinite Mixture Prototypes for Few-Shot Learning](https://mlsanity.com/paper/1902.04552/) (2019)
- [ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs](https://mlsanity.com/paper/1902.10298/) (2019)
- [ES-MAML: Simple Hessian-Free Meta Learning](https://mlsanity.com/paper/1910.01215/) (2019)
- [Meta-Learning Priors for Efficient Online Bayesian Regression](https://mlsanity.com/paper/1807.08912/) (2018)
- [The Advantage of Conditional Meta-Learning for Biased Regularization and Fine-Tuning](https://mlsanity.com/paper/2008.10857/) (2020)

## Built on by (most cited first, 5 of 5)

- [Functional Bilevel Optimization for Machine Learning](https://mlsanity.com/paper/2403.20233/) (2024)
- [Continuous-Time Meta-Learning with Forward Mode Differentiation](https://mlsanity.com/paper/2203.01443/) (2022)
- [Meta Learning of Interface Conditions for Multi-Domain Physics-Informed Neural Networks](https://mlsanity.com/paper/2210.12669/) (2022)
- [FedMeNF: Privacy-Preserving Federated Meta-Learning for Neural Fields](https://mlsanity.com/paper/2508.06301/) (2025)
- [Meta-Learning for Rapid Adaptation in Reference Tracking of Uncertain Nonlinear Systems](https://mlsanity.com/paper/2605.22513/) (2026)

## Abstract (arXiv)

Model Agnostic Meta Learning (MAML) is widely used to find a good initialization for a family of tasks. Despite its success, a critical challenge in MAML is to calculate the gradient w.r.t. the initialization of a long training trajectory for the sampled tasks, because the computation graph can rapidly explode and the computational cost is very expensive. To address this problem, we propose Adjoint MAML (A-MAML). We view gradient descent in the inner optimization as the evolution of an Ordinary Differential Equation (ODE). To efficiently compute the gradient of the validation loss w.r.t. the initialization, we use the adjoint method to construct a companion, backward ODE. To obtain the gradient w.r.t. the initialization, we only need to run the standard ODE solver twice -- one is forward in time that evolves a long trajectory of gradient flow for the sampled task; the other is backward and solves the adjoint ODE. We need not create or expand any intermediate computational graphs, adopt aggressive approximations, or impose proximal regularizers in the training loss. Our approach is cheap, accurate, and adaptable to different trajectory lengths. We demonstrate the advantage of our approach in both synthetic and real-world meta-learning tasks.