# Bootstrapped Meta-Learning

arXiv: https://arxiv.org/abs/2109.04504 · HTML: https://mlsanity.com/paper/2109.04504/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 75 references that are in the map.
- First posted on arXiv 9 Sep 2021.

## Builds on (most cited first, 30 of 47)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [Meta-Learning with Latent Embedding Optimization](https://mlsanity.com/paper/1807.05960/) (2018)
- [Policy Distillation](https://mlsanity.com/paper/1511.06295/) (2015)
- [Learning to reinforcement learn](https://mlsanity.com/paper/1611.05763/) (2016)
- [Gradient-based Hyperparameter Optimization through Reversible Learning](https://mlsanity.com/paper/1502.03492/) (2015)
- [Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML](https://mlsanity.com/paper/1909.09157/) (2019)
- [Maximum a Posteriori Policy Optimisation](https://mlsanity.com/paper/1806.06920/) (2018)
- [Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples](https://mlsanity.com/paper/1903.03096/) (2019)
- [How to train your MAML](https://mlsanity.com/paper/1810.09502/) (2018)
- [Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents](https://mlsanity.com/paper/1709.06009/) (2017)
- [Distral: Robust Multitask Reinforcement Learning](https://mlsanity.com/paper/1707.04175/) (2017)
- [Adaptive Gradient-Based Meta-Learning Methods](https://mlsanity.com/paper/1906.02717/) (2019)
- [Recasting Gradient-Based Meta-Learning as Hierarchical Bayes](https://mlsanity.com/paper/1801.08930/) (2018)
- [Revisiting Natural Gradient for Deep Networks](https://mlsanity.com/paper/1301.3584/) (2013)
- [Meta-Gradient Reinforcement Learning](https://mlsanity.com/paper/1805.09801/) (2018)
- [AutoML-Zero: Evolving Machine Learning Algorithms From Scratch](https://mlsanity.com/paper/2003.03384/) (2020)
- [Learned Optimizers that Scale and Generalize](https://mlsanity.com/paper/1703.04813/) (2017)
- [Mirror Descent Policy Optimization](https://mlsanity.com/paper/2005.09814/) (2020)
- [On Learning Intrinsic Rewards for Policy Gradient Methods](https://mlsanity.com/paper/1804.06459/) (2018)
- [Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learning](https://mlsanity.com/paper/2004.14646/) (2020)

## Built on by (most cited first, 30 of 45)

- [In-context Reinforcement Learning with Algorithm Distillation](https://mlsanity.com/paper/2210.14215/) (2022)
- [Human-Timescale Adaptation in an Open-Ended Task Space](https://mlsanity.com/paper/2301.07608/) (2023)
- [Automated Reinforcement Learning (AutoRL): A Survey and Open Problems](https://mlsanity.com/paper/2201.03916/) (2022)
- [Towards Omni-generalizable Neural Methods for Vehicle Routing Problems](https://mlsanity.com/paper/2305.19587/) (2023)
- [Hyperparameters in Reinforcement Learning and How To Tune Them](https://mlsanity.com/paper/2306.01324/) (2023)
- [A Survey of Temporal Credit Assignment in Deep Reinforcement Learning](https://mlsanity.com/paper/2312.01072/) (2023)
- [Discovering Evolution Strategies via Meta-Black-Box Optimization](https://mlsanity.com/paper/2211.11260/) (2022)
- [Continual Learning as Computationally Constrained Reinforcement Learning](https://mlsanity.com/paper/2307.04345/) (2023)
- [Meta-Learning Sparse Compression Networks](https://mlsanity.com/paper/2205.08957/) (2022)
- [Bayesian Generational Population-Based Training](https://mlsanity.com/paper/2207.09405/) (2022)
- [ReLOAD: Reinforcement Learning with Optimistic Ascent-Descent for Last-Iterate Convergence in Constrained MDPs](https://mlsanity.com/paper/2302.01275/) (2023)
- [Prediction and Control in Continual Reinforcement Learning](https://mlsanity.com/paper/2312.11669/) (2023)
- [In-context Exploration-Exploitation for Reinforcement Learning](https://mlsanity.com/paper/2403.06826/) (2024)
- [B2Opt: Learning to Optimize Black-box Optimization with Little Budget](https://mlsanity.com/paper/2304.11787/) (2023)
- [Effective Structured Prompting by Meta-Learning and Representative Verbalizer](https://mlsanity.com/paper/2306.00618/) (2023)
- [Learning Large-scale Neural Fields via Context Pruned Meta-Learning](https://mlsanity.com/paper/2302.00617/) (2023)
- [Meta-Gradients in Non-Stationary Environments](https://mlsanity.com/paper/2209.06159/) (2022)
- [Position: Lifetime tuning is incompatible with continual reinforcement learning](https://mlsanity.com/paper/2404.02113/) (2024)
- [Reactive Exploration to Cope with Non-Stationarity in Lifelong Reinforcement Learning](https://mlsanity.com/paper/2207.05742/) (2022)
- [Meta-Learning with Self-Improving Momentum Target](https://mlsanity.com/paper/2210.05185/) (2022)
- [Deep Model Predictive Optimization](https://mlsanity.com/paper/2310.04590/) (2023)
- [One Step at a Time: Pros and Cons of Multi-Step Meta-Gradient Reinforcement Learning](https://mlsanity.com/paper/2111.00206/) (2021)
- [An Investigation of the Bias-Variance Tradeoff in Meta-Gradients](https://mlsanity.com/paper/2209.11303/) (2022)
- [RotoGBML: Towards Out-of-Distribution Generalization for Gradient-Based Meta-Learning](https://mlsanity.com/paper/2303.06679/) (2023)
- [Learning to Learn Weight Generation via Local Consistency Diffusion](https://mlsanity.com/paper/2502.01117/) (2025)
- [Unleashing the Power of Meta-tuning for Few-shot Generalization Through Sparse Interpolated Experts](https://mlsanity.com/paper/2403.08477/) (2024)
- [Adaptive Q-Network: On-the-fly Target Selection for Deep Reinforcement Learning](https://mlsanity.com/paper/2405.16195/) (2024)
- [ARLBench: Flexible and Efficient Benchmarking for Hyperparameter Optimization in Reinforcement Learning](https://mlsanity.com/paper/2409.18827/) (2024)
- [Optimistic Meta-Gradients](https://mlsanity.com/paper/2301.03236/) (2023)
- [EMO: Episodic Memory Optimization for Few-Shot Meta-Learning](https://mlsanity.com/paper/2306.05189/) (2023)

## Abstract (arXiv)

Meta-learning empowers artificial intelligence to increase its efficiency by learning how to learn. Unlocking this potential involves overcoming a challenging meta-optimisation problem. We propose an algorithm that tackles this problem by letting the meta-learner teach itself. The algorithm first bootstraps a target from the meta-learner, then optimises the meta-learner by minimising the distance to that target under a chosen (pseudo-)metric. Focusing on meta-learning with gradients, we establish conditions that guarantee performance improvements and show that the metric can control meta-optimisation. Meanwhile, the bootstrapping mechanism can extend the effective meta-learning horizon without requiring backpropagation through all updates. We achieve a new state-of-the art for model-free agents on the Atari ALE benchmark and demonstrate that it yields both performance and efficiency gains in multi-task meta-learning. Finally, we explore how bootstrapping opens up new possibilities and find that it can meta-learn efficient exploration in an epsilon-greedy Q-learning agent, without backpropagating through the update rule.