# Reward Estimation for Variance Reduction in Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/1805.03359 · HTML: https://mlsanity.com/paper/1805.03359/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 May 2018.

## Built on by (most cited first, 17 of 17)

- [Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizability](https://mlsanity.com/paper/2209.08025/) (2022)
- [Sample Efficient Deep Reinforcement Learning via Uncertainty Estimation](https://mlsanity.com/paper/2201.01666/) (2022)
- [RAGEN-2: Reasoning Collapse in Agentic RL](https://mlsanity.com/paper/2604.06268/) (2026)
- [Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learning](https://mlsanity.com/paper/2006.16785/) (2020)
- [Adaptive Layer Splitting for Wireless LLM Inference in Edge Computing: A Model-Based Reinforcement Learning Approach](https://mlsanity.com/paper/2406.02616/) (2024)
- [Reinforcement Learning with Stochastic Reward Machines](https://mlsanity.com/paper/2510.14837/) (2025)
- [Distributional Reward Estimation for Effective Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/2210.07636/) (2022)
- [Disturbing Reinforcement Learning Agents with Corrupted Rewards](https://mlsanity.com/paper/2102.06587/) (2021)
- [Mutation Testing of Deep Reinforcement Learning Based on Real Faults](https://mlsanity.com/paper/2301.05651/) (2023)
- [Beyond Expected Return: Accounting for Policy Reproducibility when Evaluating Reinforcement Learning Algorithms](https://mlsanity.com/paper/2312.07178/) (2023)
- [Policy Learning and Evaluation with Randomized Quasi-Monte Carlo](https://mlsanity.com/paper/2202.07808/) (2022)
- [Actively Learning Costly Reward Functions for Reinforcement Learning](https://mlsanity.com/paper/2211.13260/) (2022)
- [Stable deep reinforcement learning method by predicting uncertainty in rewards as a subtask](https://mlsanity.com/paper/2101.06906/) (2021)
- [Robot See, Robot Do: Imitation Reward for Noisy Financial Environments](https://mlsanity.com/paper/2411.08637/) (2024)
- [Optimizing Heat Alert Issuance with Reinforcement Learning](https://mlsanity.com/paper/2312.14196/) (2023)
- [The Distributional Reward Critic Framework for Reinforcement Learning Under Perturbed Rewards](https://mlsanity.com/paper/2401.05710/) (2024)
- [Supervised Learning-enhanced Multi-Group Actor Critic for Live Stream Allocation in Feed](https://mlsanity.com/paper/2412.10381/) (2024)

## Abstract (arXiv)

Reinforcement Learning (RL) agents require the specification of a reward signal for learning behaviours. However, introduction of corrupt or stochastic rewards can yield high variance in learning. Such corruption may be a direct result of goal misspecification, randomness in the reward signal, or correlation of the reward with external factors that are not known to the agent. Corruption or stochasticity of the reward signal can be especially problematic in robotics, where goal specification can be particularly difficult for complex tasks. While many variance reduction techniques have been studied to improve the robustness of the RL process, handling such stochastic or corrupted reward structures remains difficult. As an alternative for handling this scenario in model-free RL methods, we suggest using an estimator for both rewards and value functions. We demonstrate that this improves performance under corrupted stochastic rewards in both the tabular and non-linear function approximation settings for a variety of noise types and environments. The use of reward estimation is a robust and easy-to-implement improvement for handling corrupted reward signals in model-free RL.