# From Past to Future: Rethinking Eligibility Traces

arXiv: https://arxiv.org/abs/2312.12972 · HTML: https://mlsanity.com/paper/2312.12972/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 16 references that are in the map.
- First posted on arXiv 20 Dec 2023.

## Builds on (most cited first, 10 of 10)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [Reinforcement Learning with Unsupervised Auxiliary Tasks](https://mlsanity.com/paper/1611.05397/) (2016)
- [Offline Reinforcement Learning with Reverse Model-based Imagination](https://mlsanity.com/paper/2110.00188/) (2021)
- [Expected Eligibility Traces](https://mlsanity.com/paper/2007.01839/) (2020)
- [Transforming Cooling Optimization for Green Data Center via Deep Reinforcement Learning](https://mlsanity.com/paper/1709.05077/) (2017)
- [Forethought and Hindsight in Credit Assignment](https://mlsanity.com/paper/2010.13685/) (2020)
- [Learning Retrospective Knowledge with Reverse Reinforcement Learning](https://mlsanity.com/paper/2007.06703/) (2020)
- [Robust Imitation of a Few Demonstrations with a Backwards Model](https://mlsanity.com/paper/2210.09337/) (2022)
- [What makes useful auxiliary tasks in reinforcement learning: investigating the effect of the target policy](https://mlsanity.com/paper/2204.00565/) (2022)

## Built on by (most cited first, 3 of 3)

- [Beyond Uniform Credit Assignment: Selective Eligibility Traces for RLVR](https://mlsanity.com/paper/2605.05965/) (2026)
- [Demystifying the Recency Heuristic in Temporal-Difference Learning](https://mlsanity.com/paper/2406.12284/) (2024)
- [Self-Evidencing Through Hierarchical Gradient Decomposition: A Dissipative System That Maintains Non-Equilibrium Steady-State by Minimizing Variational Free Energy](https://mlsanity.com/paper/2510.17916/) (2025)

## Abstract (arXiv)

In this paper, we introduce a fresh perspective on the challenges of credit assignment and policy evaluation. First, we delve into the nuances of eligibility traces and explore instances where their updates may result in unexpected credit assignment to preceding states. From this investigation emerges the concept of a novel value function, which we refer to as the bidirectional value function. Unlike traditional state value functions, bidirectional value functions account for both future expected returns (rewards anticipated from the current state onward) and past expected returns (cumulative rewards from the episode's start to the present). We derive principled update equations to learn this value function and, through experimentation, demonstrate its efficacy in enhancing the process of policy evaluation. In particular, our results indicate that the proposed learning approach can, in certain challenging contexts, perform policy evaluation more rapidly than TD(λ) -- a method that learns forward value functions, vπ, directly. Overall, our findings present a new perspective on eligibility traces and potential advantages associated with the novel value function it inspires, especially for policy evaluation.