mlsanityData to 7 Oct 2026

Meta-Gradient Reinforcement Learning with an Objective Discovered Online

arXiv 2007.08433 ↗First posted 16 Jul 2020cs.LG cs.AI stat.MLcited by 52

Not on the map. Why?

Abstract (arXiv)

Deep reinforcement learning includes a broad family of algorithms that parameterise an internal representation, such as a value function or policy, by a deep neural network. Each algorithm optimises its parameters with respect to an objective, such as Q-learning or policy gradient, that defines its semantics. In this work, we propose an algorithm based on meta-gradient descent that discovers its own objective, flexibly parameterised by a deep neural network, solely from interactive experience with its environment. Over time, this allows the agent to learn how to learn increasingly effectively. Furthermore, because the objective is discovered online, it can adapt to changes over time. We demonstrate that the algorithm discovers how to address several important issues in RL, such as bootstrapping, non-stationarity, and off-policy learning. On the Atari Learning Environment, the meta-gradient algorithm adapts over time to learn with greater efficiency, eventually outperforming the median score of a strong actor-critic baseline.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 17; 2022: 12; 2023: 10; 2024: 5; 2025: 8.

52 papers.

Sort:

By area:

  1. 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 115
  2. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  3. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 69
  4. 2023Hyperparameters in Reinforcement Learning and How To Tune Themcited by 58
  5. 2021Offline Meta-Reinforcement Learning with Online Self-Supervisioncited by 52
  6. 2021Meta-Learning with Task-Adaptive Loss Function for Few-Shot Learningcited by 49
  7. 2022Discovering Evolution Strategies via Meta-Black-Box Optimizationcited by 44
  8. 2021Podracer architectures for scalable Reinforcement Learningcited by 37
  9. 2023Discovering Attention-Based Genetic Algorithms via Meta-Black-Box Optimizationcited by 33
  10. 2021Introducing Symmetries to Black Box Meta Reinforcement Learningcited by 25
  11. 2021Learning Pessimism for Robust and Efficient Off-Policy Reinforcement Learningcited by 18
  12. 2022On the Effectiveness of Fine-tuning Versus Meta-reinforcement Learningcited by 18
38 more
  1. 2022Hypernetworks for Zero-shot Transfer in Reinforcement Learningcited by 18
  2. 2023AutoCost: Evolving Intrinsic Cost for Zero-violation Reinforcement Learningcited by 16
  3. 2023Discovering General Reinforcement Learning Algorithms with Adversarial Environment Designcited by 16
  4. 2023Behavior Alignment via Reward Function Optimizationcited by 16
  5. 2024In-context Exploration-Exploitation for Reinforcement Learningcited by 16
  6. 2021Training Learned Optimizers with Randomly Initialized Learned Optimizerscited by 13
  7. 2021Automated Deep Learning: Neural Architecture Search Is Not the Endcited by 13
  8. 2023Thinker: Learning to Plan and Actcited by 13
  9. 2022Reinforcement Learning with Automated Auxiliary Loss Searchcited by 12
  10. 2023Learning to Optimize for Reinforcement Learningcited by 12
  11. 2021Deep Reinforcement Learning Policies Learn Shared Adversarial Features Across MDPscited by 11
  12. 2021A Theoretical Understanding of Gradient Bias in Meta-Reinforcement Learningcited by 11
  13. 2022Meta-Gradients in Non-Stationary Environmentscited by 11
  14. 2022Automated Reinforcement Learning: An Overviewcited by 9
  15. 2021Measuring Progress in Deep Reinforcement Learning Sample Efficiencycited by 8
  16. 2024A Survey Analyzing Generalization in Deep Reinforcement Learningcited by 8
  17. 2024Step-size Optimization for Continual Learningcited by 6
  18. 2025Zero-Shot Reinforcement Learning Under Partial Observabilitycited by 6
  19. 2021Unifying Gradient Estimators for Meta-Reinforcement Learning via Off-Policy Evaluationcited by 5
  20. 2021One Step at a Time: Pros and Cons of Multi-Step Meta-Gradient Reinforcement Learningcited by 5
  21. 2021Biased Gradient Estimate with Drastic Variance Reduction for Meta Reinforcement Learningcited by 5
  22. 2022An Investigation of the Bias-Variance Tradeoff in Meta-Gradientscited by 4
  23. 2021Adaptive Pairwise Weights for Temporal Credit Assignmentcited by 2
  24. 2021Neural Auto-Curriculacited by 2
  25. 2022Auxiliary task discovery through generate-and-testcited by 2
  26. 2023Optimistic Meta-Gradientscited by 2
  27. 2025Discovering Quality-Diversity Algorithms via Meta-Black-Box Optimizationcited by 2
  28. 2022Evolving Pareto-Optimal Actor-Critic Algorithms for Generalizability and Stabilitycited by 1
  29. 2024Black Box Meta-Learning Intrinsic Rewardscited by 1
  30. 2025Outlook Towards Deployable Continual Learning for Particle Acceleratorscited by 1
  31. 2025Combining Bayesian Inference and Reinforcement Learning for Agent Decision Making: A Reviewcited by 1
  32. 2025Differentiable Evolutionary Reinforcement Learningcited by 1
  33. 2022Model Based Meta Learning of Critics for Policy Gradientscited by 0
  34. 2022Debiasing Meta-Gradient Reinforcement Learning by Learning the Outer Value Functioncited by 0
  35. 2023Acceleration in Policy Optimizationcited by 0
  36. 2023Diagnosing and exploiting the computational demands of videos games for deep reinforcement learningcited by 0
  37. 2024Dynamic Learning Rate for Deep Reinforcement Learning: A Bandit Approachcited by 0
  38. 2025Enhancing Online Reinforcement Learning with Meta-Learned Objective from Offline Datacited by 0

50 most cited of 52 All 52 (JSON)