# Smoothing Advantage Learning

arXiv: https://arxiv.org/abs/2203.10445 · HTML: https://mlsanity.com/paper/2203.10445/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 29 references that are in the map.
- First posted on arXiv 20 Mar 2022.

## Builds on (most cited first, 11 of 11)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Maxmin Q-learning: Controlling the Estimation Bias of Q-learning](https://mlsanity.com/paper/2002.06487/) (2020)
- [Diagnosing Bottlenecks in Deep Q-learning Algorithms](https://mlsanity.com/paper/1902.10250/) (2019)
- [Munchausen Reinforcement Learning](https://mlsanity.com/paper/2007.14430/) (2020)
- [Increasing the Action Gap: New Operators for Reinforcement Learning](https://mlsanity.com/paper/1512.04860/) (2015)
- [MinAtar: An Atari-Inspired Testbed for Thorough and Reproducible Reinforcement Learning Experiments](https://mlsanity.com/paper/1903.03176/) (2019)
- [Self-Imitation Advantage Learning](https://mlsanity.com/paper/2012.11989/) (2020)
- [Stabilizing Q Learning Via Soft Mellowmax Operator](https://mlsanity.com/paper/2012.09456/) (2020)
- [Unifying Value Iteration, Advantage Learning, and Dynamic Policy Programming](https://mlsanity.com/paper/1710.10866/) (2017)

## Built on by (most cited first, 1 of 1)

- [AdaMemento: Adaptive Memory-Assisted Policy Optimization for Reinforcement Learning](https://mlsanity.com/paper/2410.04498/) (2024)

## Abstract (arXiv)

Advantage learning (AL) aims to improve the robustness of value-based reinforcement learning against estimation errors with action-gap-based regularization. Unfortunately, the method tends to be unstable in the case of function approximation. In this paper, we propose a simple variant of AL, named smoothing advantage learning (SAL), to alleviate this problem. The key to our method is to replace the original Bellman Optimal operator in AL with a smooth one so as to obtain more reliable estimation of the temporal difference target. We give a detailed account of the resulting action gap and the performance bound for approximate SAL. Further theoretical analysis reveals that the proposed value smoothing technique not only helps to stabilize the training procedure of AL by controlling the trade-off between convergence rate and the upper bound of the approximation errors, but is beneficial to increase the action gap between the optimal and sub-optimal action value as well.