# Selective Credit Assignment

arXiv: https://arxiv.org/abs/2202.09699 · HTML: https://mlsanity.com/paper/2202.09699/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 53 references that are in the map.
- First posted on arXiv 20 Feb 2022.

## Builds on (most cited first, 23 of 23)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [Deep Reinforcement Learning and the Deadly Triad](https://mlsanity.com/paper/1812.02648/) (2018)
- [RUDDER: Return Decomposition for Delayed Rewards](https://mlsanity.com/paper/1806.07857/) (2018)
- [Meta-Gradient Reinforcement Learning](https://mlsanity.com/paper/1805.09801/) (2018)
- [When to use parametric models in reinforcement learning?](https://mlsanity.com/paper/1906.05243/) (2019)
- [An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learning](https://mlsanity.com/paper/1503.04269/) (2015)
- [Hindsight Credit Assignment](https://mlsanity.com/paper/1912.02503/) (2019)
- [DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction](https://mlsanity.com/paper/2003.07305/) (2020)
- [Counterfactual Credit Assignment in Model-Free Reinforcement Learning](https://mlsanity.com/paper/2011.09464/) (2020)
- [Optimizing Agent Behavior over Long Time Scales by Transporting Value](https://mlsanity.com/paper/1810.06721/) (2018)
- [Unifying task specification in reinforcement learning](https://mlsanity.com/paper/1609.01995/) (2016)
- [Sparse Attentive Backtracking: Temporal CreditAssignment Through Reminding](https://mlsanity.com/paper/1809.03702/) (2018)
- [Expected Eligibility Traces](https://mlsanity.com/paper/2007.01839/) (2020)
- [The Termination Critic](https://mlsanity.com/paper/1902.09996/) (2019)
- [On Inductive Biases in Deep Reinforcement Learning](https://mlsanity.com/paper/1907.02908/) (2019)
- [A Greedy Approach to Adapting the Trace Parameter for Temporal Difference Learning](https://mlsanity.com/paper/1607.00446/) (2016)
- [Generalized Emphatic Temporal Difference Learning: Bias-Variance Analysis](https://mlsanity.com/paper/1509.05172/) (2015)
- [Forethought and Hindsight in Credit Assignment](https://mlsanity.com/paper/2010.13685/) (2020)
- [Learning Retrospective Knowledge with Reverse Reinforcement Learning](https://mlsanity.com/paper/2007.06703/) (2020)
- [Preferential Temporal Difference Learning](https://mlsanity.com/paper/2106.06508/) (2021)
- [Learning to Predict Independent of Span](https://mlsanity.com/paper/1508.04582/) (2015)
- [Learning Expected Emphatic Traces for Deep RL](https://mlsanity.com/paper/2107.05405/) (2021)

## Built on by (most cited first, 4 of 4)

- [A Survey of Temporal Credit Assignment in Deep Reinforcement Learning](https://mlsanity.com/paper/2312.01072/) (2023)
- [Beyond Uniform Credit Assignment: Selective Eligibility Traces for RLVR](https://mlsanity.com/paper/2605.05965/) (2026)
- [Hindsight-DICE: Stable Credit Assignment for Deep Reinforcement Learning](https://mlsanity.com/paper/2307.11897/) (2023)
- [Discerning Temporal Difference Learning](https://mlsanity.com/paper/2310.08091/) (2023)

## Abstract (arXiv)

Efficient credit assignment is essential for reinforcement learning algorithms in both prediction and control settings. We describe a unified view on temporal-difference algorithms for selective credit assignment. These selective algorithms apply weightings to quantify the contribution of learning updates. We present insights into applying weightings to value-based learning and planning algorithms, and describe their role in mediating the backward credit distribution in prediction and control. Within this space, we identify some existing online learning algorithms that can assign credit selectively as special cases, as well as add new algorithms that assign credit backward in time counterfactually, allowing credit to be assigned off-trajectory and off-policy.