# Forethought and Hindsight in Credit Assignment

arXiv: https://arxiv.org/abs/2010.13685 · HTML: https://mlsanity.com/paper/2010.13685/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Oct 2020.

## Built on by (most cited first, 16 of 16)

- [Sample-Efficient Multi-Objective Learning via Generalized Policy Improvement Prioritization](https://mlsanity.com/paper/2301.07784/) (2023)
- [Offline Reinforcement Learning with Reverse Model-based Imagination](https://mlsanity.com/paper/2110.00188/) (2021)
- [Survey of Recent Multi-Agent Reinforcement Learning Algorithms Utilizing Centralized Training](https://mlsanity.com/paper/2107.14316/) (2021)
- [Goal-Space Planning with Subgoal Models](https://mlsanity.com/paper/2206.02902/) (2022)
- [A Unified View on Solving Objective Mismatch in Model-Based Reinforcement Learning](https://mlsanity.com/paper/2310.06253/) (2023)
- [Self-Consistent Models and Values](https://mlsanity.com/paper/2110.12840/) (2021)
- [Preferential Temporal Difference Learning](https://mlsanity.com/paper/2106.06508/) (2021)
- [No More Pesky Hyperparameters: Offline Hyperparameter Tuning for RL](https://mlsanity.com/paper/2205.08716/) (2022)
- [Selective Credit Assignment](https://mlsanity.com/paper/2202.09699/) (2022)
- [From Past to Future: Rethinking Eligibility Traces](https://mlsanity.com/paper/2312.12972/) (2023)
- [Backward Learning for Goal-Conditioned Policies](https://mlsanity.com/paper/2312.05044/) (2023)
- [Sequence Compression Speeds Up Credit Assignment in Reinforcement Learning](https://mlsanity.com/paper/2405.03878/) (2024)
- [Why long model-based rollouts are no reason for bad Q-value estimates](https://mlsanity.com/paper/2407.11751/) (2024)
- [A Generalized Bootstrap Target for Value-Learning, Efficiently Combining Value and Feature Predictions](https://mlsanity.com/paper/2201.01836/) (2022)
- [Diagnosing and exploiting the computational demands of videos games for deep reinforcement learning](https://mlsanity.com/paper/2309.13181/) (2023)
- [A New View on Planning in Online Reinforcement Learning](https://mlsanity.com/paper/2406.01562/) (2024)

## Abstract (arXiv)

We address the problem of credit assignment in reinforcement learning and explore fundamental questions regarding the way in which an agent can best use additional computation to propagate new information, by planning with internal models of the world to improve its predictions. Particularly, we work to understand the gains and peculiarities of planning employed as forethought via forward models or as hindsight operating with backward models. We establish the relative merits, limitations and complementary properties of both planning mechanisms in carefully constructed scenarios. Further, we investigate the best use of models in planning, primarily focusing on the selection of states in which predictions should be (re)-evaluated. Lastly, we discuss the issue of model estimation and highlight a spectrum of methods that stretch from explicit environment-dynamics predictors to more abstract planner-aware models.