# Robust Action Gap Increasing with Clipped Advantage Learning

arXiv: https://arxiv.org/abs/2203.11677 · HTML: https://mlsanity.com/paper/2203.11677/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 22 references that are in the map.
- First posted on arXiv 20 Mar 2022.

## Builds on (most cited first, 13 of 13)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [Distributed Prioritized Experience Replay](https://mlsanity.com/paper/1803.00933/) (2018)
- [Safe and Efficient Off-Policy Reinforcement Learning](https://mlsanity.com/paper/1606.02647/) (2016)
- [Munchausen Reinforcement Learning](https://mlsanity.com/paper/2007.14430/) (2020)
- [Increasing the Action Gap: New Operators for Reinforcement Learning](https://mlsanity.com/paper/1512.04860/) (2015)
- [Self-Imitation Learning by Planning](https://mlsanity.com/paper/2103.13834/) (2021)
- [Using a Logarithmic Mapping to Enable Lower Discount Factors in Reinforcement Learning](https://mlsanity.com/paper/1906.00572/) (2019)
- [Self-Imitation Advantage Learning](https://mlsanity.com/paper/2012.11989/) (2020)
- [Unifying Value Iteration, Advantage Learning, and Dynamic Policy Programming](https://mlsanity.com/paper/1710.10866/) (2017)
- [Gap-Increasing Policy Evaluation for Efficient and Noise-Tolerant Reinforcement Learning](https://mlsanity.com/paper/1906.07586/) (2019)

## Built on by (most cited first, 1 of 1)

- [Mirror Descent Actor Critic via Bounded Advantage Learning](https://mlsanity.com/paper/2502.03854/) (2025)

## Abstract (arXiv)

Advantage Learning (AL) seeks to increase the action gap between the optimal action and its competitors, so as to improve the robustness to estimation errors. However, the method becomes problematic when the optimal action induced by the approximated value function does not agree with the true optimal action. In this paper, we present a novel method, named clipped Advantage Learning (clipped AL), to address this issue. The method is inspired by our observation that increasing the action gap blindly for all given samples while not taking their necessities into account could accumulate more errors in the performance loss bound, leading to a slow value convergence, and to avoid that, we should adjust the advantage value adaptively. We show that our simple clipped AL operator not only enjoys fast convergence guarantee but also retains proper action gaps, hence achieving a good balance between the large action gap and the fast convergence. The feasibility and effectiveness of the proposed method are verified empirically on several RL benchmarks with promising performance.