mlsanityData to 7 Oct 2026

Hindsight Foresight Relabeling for Meta-Reinforcement Learning

arXiv 2109.09031 ↗First posted 18 Sep 2021cs.LG stat.MLcited by 4

Not on the map. Why?

Abstract (arXiv)

Meta-reinforcement learning (meta-RL) algorithms allow for agents to learn new behaviors from small amounts of experience, mitigating the sample inefficiency problem in RL. However, while meta-RL agents can adapt quickly to new tasks at test time after experiencing only a few trajectories, the meta-training process is still sample-inefficient. Prior works have found that in the multi-task RL setting, relabeling past transitions and thus sharing experience among tasks can improve sample efficiency and asymptotic performance. We apply this idea to the meta-RL setting and devise a new relabeling method called Hindsight Foresight Relabeling (HFR). We construct a relabeling distribution using the combination of "hindsight", which is used to relabel trajectories using reward functions from the training task distribution, and "foresight", which takes the relabeled trajectories and computes the utility of each trajectory for each task. HFR is easy to implement and readily compatible with existing meta-RL algorithms. We find that HFR improves performance when compared to other relabeling methods on a variety of meta-RL tasks.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 21 of its 40 references that are in the map. First posted on arXiv 18 Sep 2021.

Builds on

21 of 40 references.

References · 21

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  5. 2017Hindsight Experience Replaycited by 1,096
  6. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
  7. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
  8. 2016Learning to reinforcement learncited by 369
  9. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 210
  10. 2018Meta-Gradient Reinforcement Learningcited by 156
  11. 2018Meta-Reinforcement Learning of Structured Exploration Strategiescited by 128
  12. 2018ProMP: Proximal Meta-Policy Searchcited by 91
9 more
  1. 2018Differentiable plasticity: training plastic neural networks with backpropagationcited by 83
  2. 2019Meta-Q-Learningcited by 62
  3. 2020Rewriting History with Inverse RL: Hindsight Inference for Policy Improvementcited by 59
  4. 2020Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrificescited by 59
  5. 2020Generalized Hindsight for Reinforcement Learningcited by 51
  6. 2018Some Considerations on Learning to Explore via Meta-Reinforcement Learningcited by 43
  7. 2019What Can Learned Intrinsic Rewards Capture?cited by 36
  8. 2020Model-based Adversarial Meta-Reinforcement Learningcited by 32
  9. 2020Meta-Reinforcement Learning Robust to Distributional Shift via Model Identification and Experience Relabelingcited by 24

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 0; 2025: 3.

4 papers.

  1. 2023Design from Policies: Conservative Test-Time Adaptation for Offline Policy Optimizationcited by 8
  2. 2025Self-Improving Skill Learning for Robust Skill-based Meta-Reinforcement Learningcited by 5
  3. 2025Task-Aware Virtual Training: Enhancing Generalization in Meta-Reinforcement Learning for Out-of-Distribution Taskscited by 3
  4. 2025Combining Bayesian Inference and Reinforcement Learning for Agent Decision Making: A Reviewcited by 1