# Learning Guidance Rewards with Trajectory-space Smoothing

arXiv: https://arxiv.org/abs/2010.12718 · HTML: https://mlsanity.com/paper/2010.12718/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Oct 2020.

## Built on by (most cited first, 30 of 30)

- [Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs](https://mlsanity.com/paper/2110.05038/) (2021)
- [Preference Transformer: Modeling Human Preferences using Transformers for RL](https://mlsanity.com/paper/2303.00957/) (2023)
- [Dense Reward for Free in Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2402.00782/) (2024)
- [Learning Long-Term Reward Redistribution via Randomized Return Decomposition](https://mlsanity.com/paper/2111.13485/) (2021)
- [Off-Policy Reinforcement Learning with Delayed Rewards](https://mlsanity.com/paper/2106.11854/) (2021)
- [Learning Energy Decompositions for Partial Inference of GFlowNets](https://mlsanity.com/paper/2310.03301/) (2023)
- [Interpretable Reward Redistribution in Reinforcement Learning: A Causal Approach](https://mlsanity.com/paper/2305.18427/) (2023)
- [Provably Efficient Offline Reinforcement Learning with Trajectory-Wise Reward](https://mlsanity.com/paper/2206.06426/) (2022)
- [Agent-Temporal Attention for Reward Redistribution in Episodic Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2201.04612/) (2022)
- [Neural Episodic Control with State Abstraction](https://mlsanity.com/paper/2301.11490/) (2023)
- [STAS: Spatial-Temporal Return Decomposition for Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2304.07520/) (2023)
- [Sentence-level Reward Model can Generalize Better for Aligning LLM from Human Preference](https://mlsanity.com/paper/2503.04793/) (2025)
- [Attention-Based Reward Shaping for Sparse and Delayed Rewards](https://mlsanity.com/paper/2505.10802/) (2025)
- [Multi-Agent Advisor Q-Learning](https://mlsanity.com/paper/2111.00345/) (2021)
- [Hindsight-DICE: Stable Credit Assignment for Deep Reinforcement Learning](https://mlsanity.com/paper/2307.11897/) (2023)
- [Shaping Advice in Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2103.15941/) (2021)
- [Episodic Return Decomposition by Difference of Implicitly Assigned Sub-Trajectory Reward](https://mlsanity.com/paper/2312.10642/) (2023)
- [Beyond Simple Sum of Delayed Rewards: Non-Markovian Reward Modeling for Reinforcement Learning](https://mlsanity.com/paper/2410.20176/) (2024)
- [ELEMENT: Episodic and Lifelong Exploration via Maximum Entropy](https://mlsanity.com/paper/2412.03800/) (2024)
- [Aligning Dialogue Agents with Global Feedback via Large Language Model Multimodal Reward Decomposition](https://mlsanity.com/paper/2505.15922/) (2025)
- [Preference-Guided Learning for Sparse-Reward Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2509.21828/) (2025)
- [Off-Beat Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2205.13718/) (2022)
- [Improving Dialogue Agents by Decomposing One Global Explicit Annotation with Local Implicit Multimodal Feedback](https://mlsanity.com/paper/2403.11330/) (2024)
- [Agent-Temporal Credit Assignment for Optimal Policy Preservation in Sparse Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2412.14779/) (2024)
- [CoCA: Step-level Reward for Free in RL-based T2I Diffusion Model Fine-tuning](https://mlsanity.com/paper/2505.19196/) (2025)
- [PRISM: Parallel Reward Integration with Symmetry for MORL](https://mlsanity.com/paper/2602.18277/) (2026)
- [Shaping Advice in Deep Reinforcement Learning](https://mlsanity.com/paper/2202.09489/) (2022)
- [Policy Optimization with Smooth Guidance Learned from State-Only Demonstrations](https://mlsanity.com/paper/2401.00162/) (2023)
- [Reinforcement Learning from Bagged Reward](https://mlsanity.com/paper/2402.03771/) (2024)
- [Redistributing Rewards Across Time and Agents for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2502.04864/) (2025)

## Abstract (arXiv)

Long-term temporal credit assignment is an important challenge in deep reinforcement learning (RL). It refers to the ability of the agent to attribute actions to consequences that may occur after a long time interval. Existing policy-gradient and Q-learning algorithms typically rely on dense environmental rewards that provide rich short-term supervision and help with credit assignment. However, they struggle to solve tasks with delays between an action and the corresponding rewarding feedback. To make credit assignment easier, recent works have proposed algorithms to learn dense "guidance" rewards that could be used in place of the sparse or delayed environmental rewards. This paper is in the same vein -- starting with a surrogate RL objective that involves smoothing in the trajectory-space, we arrive at a new algorithm for learning guidance rewards. We show that the guidance rewards have an intuitive interpretation, and can be obtained without training any additional neural networks. Due to the ease of integration, we use the guidance rewards in a few popular algorithms (Q-learning, Actor-Critic, Distributional-RL) and present results in single-agent and multi-agent tasks that elucidate the benefit of our approach when the environmental rewards are sparse or delayed.