# Stable deep reinforcement learning method by predicting uncertainty in rewards as a subtask

arXiv: https://arxiv.org/abs/2101.06906 · HTML: https://mlsanity.com/paper/2101.06906/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 27 references that are in the map.
- First posted on arXiv 18 Jan 2021.

## Builds on (most cited first, 16 of 16)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Residual Attention Network for Image Classification](https://mlsanity.com/paper/1704.06904/) (2017)
- [Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection](https://mlsanity.com/paper/1603.02199/) (2016)
- [Attention Branch Network: Learning of Attention Mechanism for Visual Explanation](https://mlsanity.com/paper/1812.10025/) (2018)
- [Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning](https://mlsanity.com/paper/1611.01929/) (2016)
- [Reinforcement Learning with a Corrupted Reward Channel](https://mlsanity.com/paper/1705.08417/) (2017)
- [Grasp2Vec: Learning Object Representations from Self-Supervised Grasping](https://mlsanity.com/paper/1811.06964/) (2018)
- [Reward Estimation for Variance Reduction in Deep Reinforcement Learning](https://mlsanity.com/paper/1805.03359/) (2018)
- [Stochastic Variance Reduction for Deep Q-learning](https://mlsanity.com/paper/1905.08152/) (2019)
- [Motion Switching with Sensory and Instruction Signals by designing Dynamical Systems using Deep Neural Network](https://mlsanity.com/paper/1712.05109/) (2017)
- [Online Self-Supervised Learning for Object Picking: Detecting Optimum Grasping Position using a Metric Learning Approach](https://mlsanity.com/paper/2003.03717/) (2020)

## Built on by (most cited first, 1 of 1)

- ["Think Before You Speak": Improving Multi-Action Dialog Policy by Planning Single-Action Dialogs](https://mlsanity.com/paper/2204.11481/) (2022)

## Abstract (arXiv)

In recent years, a variety of tasks have been accomplished by deep reinforcement learning (DRL). However, when applying DRL to tasks in a real-world environment, designing an appropriate reward is difficult. Rewards obtained via actual hardware sensors may include noise, misinterpretation, or failed observations. The learning instability caused by these unstable signals is a problem that remains to be solved in DRL. In this work, we propose an approach that extends existing DRL models by adding a subtask to directly estimate the variance contained in the reward signal. The model then takes the feature map learned by the subtask in a critic network and sends it to the actor network. This enables stable learning that is robust to the effects of potential noise. The results of experiments in the Atari game domain with unstable reward signals show that our method stabilizes training convergence. We also discuss the extensibility of the model by visualizing feature maps. This approach has the potential to make DRL more practical for use in noisy, real-world scenarios.