mlsanityData to 7 Oct 2026

Policy Gradients Incorporating the Future

arXiv 2108.02096 ↗First posted 4 Aug 2021cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Reasoning about the future -- understanding how decisions in the present time affect outcomes in the future -- is one of the central challenges for reinforcement learning (RL), especially in highly-stochastic or partially observable environments. While predicting the future directly is hard, in this work we introduce a method that allows an agent to "look into the future" without explicitly predicting it. Namely, we propose to allow an agent, during its training on past experience, to observe what actually happened in the future at that time, while enforcing an information bottleneck to avoid the agent overly relying on this privileged information. This gives our agent the opportunity to utilize rich and useful information about the future trajectory dynamics in addition to the present. Our method, Policy Gradients Incorporating the Future (PGIF), is easy to implement and versatile, being applicable to virtually any policy gradient algorithm. We apply our proposed method to a number of off-the-shelf RL algorithms and show that PGIF is able to achieve higher reward faster in a variety of online and offline RL domains, as well as sparse-reward and partially observable environments.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 40 of its 58 references that are in the map. First posted on arXiv 4 Aug 2021.

Builds on

40 of 58 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  9. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  10. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  11. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  12. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
28 more
  1. 2015Generating Sentences from a Continuous Spacecited by 607
  2. 2019Model-Based Reinforcement Learning for Ataricited by 495
  3. 2017Count-Based Exploration with Neural Density Modelscited by 255
  4. 2019Recurrent Independent Mechanismscited by 219
  5. 2016Professor Forcing: A New Algorithm for Training Recurrent Networkscited by 214
  6. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  7. 2017Reverse Curriculum Generation for Reinforcement Learningcited by 185
  8. 2018Deep Reinforcement Learning and the Deadly Triadcited by 170
  9. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
  10. 2019Soft Actor-Critic for Discrete Action Settingscited by 161
  11. 2019Benchmarking Model-Based Reinforcement Learningcited by 158
  12. 2016Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Datacited by 154
  13. 2017Imagination-Augmented Agents for Deep Reinforcement Learningcited by 137
  14. 2019Behaviour Suite for Reinforcement Learningcited by 129
  15. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 124
  16. 2017Value Prediction Networkcited by 117
  17. 2021Representation Matters: Offline Pretraining for Sequential Decision Makingcited by 90
  18. 2013The Optimal Reward Baseline for Gradient-Based Reinforcement Learningcited by 84
  19. 2019Hindsight Credit Assignmentcited by 74
  20. 2014Learning Stochastic Recurrent Networkscited by 60
  21. 2020Counterfactual Credit Assignment in Model-Free Reinforcement Learningcited by 60
  22. 2017Z-Forcing: Training Stochastic Recurrent Networkscited by 51
  23. 2018Combined Reinforcement Learning via Abstract Representationscited by 37
  24. 2020Data-efficient Hindsight Off-policy Option Learningcited by 31
  25. 2020Selective Dyna-style Planning Under Limited Model Capacitycited by 30
  26. 2017Learning model-based planning from scratchcited by 21
  27. 2018Deep Dyna-Q: Integrating Planning for Task-Completion Dialogue Policy Learningcited by 21
  28. 2020Value-driven Hindsight Modellingcited by 6

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 2; 2023: 3.

6 papers.

  1. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 105
  2. 2022Dichotomy of Control: Separating What You Can Control from What You Cannotcited by 47
  3. 2023Future-conditioned Unsupervised Pretraining for Decision Transformercited by 26
  4. 2022Hindsight Learning for MDPs with Exogenous Inputscited by 25
  5. 2023Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysiscited by 10
  6. 2023Generative Intrinsic Optimization: Intrinsic Control with Model Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning from the Near Future: Temporal Self-Distillation for RLVR
  2. 2026FutureWorld: A Live Reinforcement Learning Environment for Predictive Agents with Real-World Outcome Rewards
  3. 2025Forecasting in Offline Reinforcement Learning for Non-stationary Environments
  4. 2025Future Policy Approximation for Offline Reinforcement Learning in LLM Reasoning
  5. 2019Learning to Predict Without Looking Ahead: World Models Without Forward Prediction
  6. 2026Prospective Hindsight: Self-Calibrating Reinforcement Learning via Prediction-Reality Gaps
  7. 2024Reinforcement Learning from Delayed Observations via World Models
  8. 2026Self-Retrospection Distillation: Turning Post-hoc Experiences into Prior Foresight