mlsanityData to 7 Oct 2026

Bootstrapped Meta-Learning

arXiv 2109.04504 ↗First posted 9 Sep 2021cs.LG cs.AI stat.MLcited by 45

Not on the map. Why?

Abstract (arXiv)

Meta-learning empowers artificial intelligence to increase its efficiency by learning how to learn. Unlocking this potential involves overcoming a challenging meta-optimisation problem. We propose an algorithm that tackles this problem by letting the meta-learner teach itself. The algorithm first bootstraps a target from the meta-learner, then optimises the meta-learner by minimising the distance to that target under a chosen (pseudo-)metric. Focusing on meta-learning with gradients, we establish conditions that guarantee performance improvements and show that the metric can control meta-optimisation. Meanwhile, the bootstrapping mechanism can extend the effective meta-learning horizon without requiring backpropagation through all updates. We achieve a new state-of-the art for model-free agents on the Atari ALE benchmark and demonstrate that it yields both performance and efficiency gains in multi-task meta-learning. Finally, we explore how bootstrapping opens up new possibilities and find that it can meta-learn efficient exploration in an epsilon-greedy Q-learning agent, without backpropagating through the update rule.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Builds on 47 of its 75 references that are in the map. First posted on arXiv 9 Sep 2021.

Builds on

47 of 75 references.

References · 47

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  6. 2015Trust Region Policy Optimizationcited by 2,503
  7. 2016Matching Networks for One Shot Learningcited by 2,246
  8. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  9. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  10. 2016Learning to learn by gradient descent by gradient descentcited by 616
  11. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 610
  12. 2018Meta-Learning with Latent Embedding Optimizationcited by 476
35 more
  1. 2015Policy Distillationcited by 414
  2. 2016Learning to reinforcement learncited by 369
  3. 2019Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAMLcited by 359
  4. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 357
  5. 2018Maximum a Posteriori Policy Optimisationcited by 318
  6. 2019Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examplescited by 312
  7. 2018How to train your MAMLcited by 306
  8. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  9. 2017Distral: Robust Multitask Reinforcement Learningcited by 231
  10. 2019Adaptive Gradient-Based Meta-Learning Methodscited by 183
  11. 2018Recasting Gradient-Based Meta-Learning as Hierarchical Bayescited by 180
  12. 2013Revisiting Natural Gradient for Deep Networkscited by 157
  13. 2018Meta-Gradient Reinforcement Learningcited by 156
  14. 2020AutoML-Zero: Evolving Machine Learning Algorithms From Scratchcited by 137
  15. 2017Learned Optimizers that Scale and Generalizecited by 109
  16. 2020Mirror Descent Policy Optimizationcited by 102
  17. 2020Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learningcited by 101
  18. 2018On Learning Intrinsic Rewards for Policy Gradient Methodscited by 100
  19. 2019Meta-Learning with Warped Gradient Descentcited by 98
  20. 2018Understanding and correcting pathologies in the training of learned optimizerscited by 97
  21. 2019Meta-Learning without Memorizationcited by 92
  22. 2020Discovering Reinforcement Learning Algorithmscited by 75
  23. 2019Meta-Curvaturecited by 70
  24. 2018Understanding Short-Horizon Bias in Stochastic Meta-Optimizationcited by 68
  25. 2019Improving Generalization in Meta Reinforcement Learning using Learned Objectivescited by 68
  26. 2019Provable Guarantees for Gradient-Based Meta-Learningcited by 67
  27. 2021Muesli: Combining Improvements in Policy Optimizationcited by 53
  28. 2017Learning Gradient Descent: Better Generalization and Longer Horizonscited by 50
  29. 2016MetaGrad: Multiple Learning Rates in Online Learningcited by 46
  30. 2018Some Considerations on Learning to Explore via Meta-Reinforcement Learningcited by 43
  31. 2019Off-Policy Actor-Critic with Shared Experience Replaycited by 39
  32. 2020Meta-learning curiosity algorithmscited by 25
  33. 2019Learning to Optimize in Swarmscited by 23
  34. 2018Transferring Knowledge across Learning Processescited by 22
  35. 2016Learning to Learn without Gradient Descent by Gradient Descentcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 13; 2023: 20; 2024: 9; 2025: 1; 2026: 1.

45 papers.

Sort:

By area:

  1. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
  2. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 136
  3. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 69
  4. 2023Towards Omni-generalizable Neural Methods for Vehicle Routing Problemscited by 68
  5. 2023Hyperparameters in Reinforcement Learning and How To Tune Themcited by 58
  6. 2023A Survey of Temporal Credit Assignment in Deep Reinforcement Learningcited by 56
  7. 2022Discovering Evolution Strategies via Meta-Black-Box Optimizationcited by 44
  8. 2023Continual Learning as Computationally Constrained Reinforcement Learningcited by 35
  9. 2022Meta-Learning Sparse Compression Networkscited by 22
  10. 2022Bayesian Generational Population-Based Trainingcited by 20
  11. 2023ReLOAD: Reinforcement Learning with Optimistic Ascent-Descent for Last-Iterate Convergence in Constrained MDPscited by 19
  12. 2023Prediction and Control in Continual Reinforcement Learningcited by 19
33 more
  1. 2024In-context Exploration-Exploitation for Reinforcement Learningcited by 16
  2. 2023B2Opt: Learning to Optimize Black-box Optimization with Little Budgetcited by 15
  3. 2023Effective Structured Prompting by Meta-Learning and Representative Verbalizercited by 14
  4. 2023Learning Large-scale Neural Fields via Context Pruned Meta-Learningcited by 12
  5. 2022Meta-Gradients in Non-Stationary Environmentscited by 11
  6. 2022Reactive Exploration to Cope with Non-Stationarity in Lifelong Reinforcement Learningcited by 9
  7. 2022Meta-Learning with Self-Improving Momentum Targetcited by 9
  8. 2024Position: Lifetime tuning is incompatible with continual reinforcement learningcited by 9
  9. 2023Deep Model Predictive Optimizationcited by 7
  10. 2021One Step at a Time: Pros and Cons of Multi-Step Meta-Gradient Reinforcement Learningcited by 5
  11. 2023RotoGBML: Towards Out-of-Distribution Generalization for Gradient-Based Meta-Learningcited by 5
  12. 2022An Investigation of the Bias-Variance Tradeoff in Meta-Gradientscited by 4
  13. 2025Learning to Learn Weight Generation via Local Consistency Diffusioncited by 4
  14. 2024Unleashing the Power of Meta-tuning for Few-shot Generalization Through Sparse Interpolated Expertscited by 3
  15. 2024Adaptive Q-Network: On-the-fly Target Selection for Deep Reinforcement Learningcited by 3
  16. 2024ARLBench: Flexible and Efficient Benchmarking for Hyperparameter Optimization in Reinforcement Learningcited by 3
  17. 2023Optimistic Meta-Gradientscited by 2
  18. 2023EMO: Episodic Memory Optimization for Few-Shot Meta-Learningcited by 2
  19. 2023Introspective Action Advising for Interpretable Transfer Learningcited by 2
  20. 2023Unleash Model Potential: Bootstrapped Meta Self-supervised Learningcited by 2
  21. 2022Reinforcement Teachingcited by 1
  22. 2023Accelerating Neural Self-Improvement via Bootstrappingcited by 1
  23. 2024Rethinking Meta-Learning from a Learning Lenscited by 1
  24. 2026Counteractive RL: Rethinking Core Principles for Efficient and Scalable Deep Reinforcement Learningcited by 1
  25. 2022Dynamic Memory-based Curiosity: A Bootstrap Approach for Explorationcited by 0
  26. 2022Debiasing Meta-Gradient Reinforcement Learning by Learning the Outer Value Functioncited by 0
  27. 2022POMRL: No-Regret Learning-to-Plan with Increasing Horizonscited by 0
  28. 2023Acceleration in Policy Optimizationcited by 0
  29. 2023Ever Evolving Evaluator (EV3): Towards Flexible and Reliable Meta-Optimization for Knowledge Distillationcited by 0
  30. 2023Successive Model-Agnostic Meta-Learning for Few-Shot Fault Time Series Prognosiscited by 0
  31. 2024Progressive Conservative Adaptation for Evolving Target Domainscited by 0
  32. 2024Communication-Efficient and Privacy-Preserving Decentralized Meta-Learningcited by 0
  33. 2024Meta-Gradient Search Control: A Method for Improving the Efficiency of Dyna-style Planningcited by 0