mlsanityData to 7 Oct 2026

Model Based Meta Learning of Critics for Policy Gradients

arXiv 2204.02210 ↗First posted 5 Apr 2022cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Being able to seamlessly generalize across different tasks is fundamental for robots to act in our world. However, learning representations that generalize quickly to new scenarios is still an open research problem in reinforcement learning. In this paper we present a framework to meta-learn the critic for gradient-based policy learning. Concretely, we propose a model-based bi-level optimization algorithm that updates the critics parameters such that the policy that is learned with the updated critic gets closer to solving the meta-training tasks. We illustrate that our algorithm leads to learned critics that resemble the ground truth Q function for a given task. Finally, after meta-training, the learned critic can be used to learn new policies for new unseen task and environment settings via model-free policy gradient optimization, without requiring a model. We present results that show the generalization capabilities of our learned critic to new tasks and dynamics when used to learn a new policy in a new scenario.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 28 of its 41 references that are in the map. First posted on arXiv 5 Apr 2022.

Builds on

28 of 41 references.

References · 28

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2017Hindsight Experience Replaycited by 1,096
  4. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  5. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  6. 2016Learning to learn by gradient descent by gradient descentcited by 616
  7. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
  8. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 357
  9. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  10. 2017Forward and Reverse Gradient-Based Hyperparameter Optimizationcited by 232
  11. 2016Learning to Optimizecited by 195
  12. 2018A Study on Overfitting in Deep Reinforcement Learningcited by 170
16 more
  1. 2018One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learningcited by 146
  2. 2018Meta-Reinforcement Learning of Structured Exploration Strategiescited by 128
  3. 2018Evolved Policy Gradientscited by 86
  4. 2018A Dissection of Overfitting and Generalization in Continuous Reinforcement Learningcited by 86
  5. 2020NeuralSim: Augmenting Differentiable Simulators with Neural Networkscited by 77
  6. 2019Meta-Learning via Learned Losscited by 60
  7. 2020Meta-Gradient Reinforcement Learning with an Objective Discovered Onlinecited by 52
  8. 2019Reward Shaping via Meta-Learningcited by 30
  9. 2017Learning to Learn: Meta-Critic Networks for Sample Efficient Learningcited by 28
  10. 2019Guided Meta-Policy Searchcited by 26
  11. 2020Encoding Physical Constraints in Differentiable Newton-Euler Algorithmcited by 23
  12. 2019Investigating Generalisation in Continuous Deep Reinforcement Learningcited by 18
  13. 2020Online Meta-Critic Learning for Off-Policy Actor-Critic Methodscited by 17
  14. 2019Curious iLQR: Resolving Uncertainty in Model-based RLcited by 11
  15. 2020Augmenting Differentiable Simulators with Neural Networks to Close the Sim2Real Gapcited by 9
  16. 2017Online Learning of a Memory for Learning Ratescited by 4

Built on by

Not cited yet.