# Gap-Increasing Policy Evaluation for Efficient and Noise-Tolerant Reinforcement Learning

arXiv: https://arxiv.org/abs/1906.07586 · HTML: https://mlsanity.com/paper/1906.07586/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jun 2019.

## Built on by (most cited first, 2 of 2)

- [VA-learning as a more efficient alternative to Q-learning](https://mlsanity.com/paper/2305.18161/) (2023)
- [Robust Action Gap Increasing with Clipped Advantage Learning](https://mlsanity.com/paper/2203.11677/) (2022)

## Abstract (arXiv)

In real-world applications of reinforcement learning (RL), noise from inherent stochasticity of environments is inevitable. However, current policy evaluation algorithms, which plays a key role in many RL algorithms, are either prone to noise or inefficient. To solve this issue, we introduce a novel policy evaluation algorithm, which we call Gap-increasing RetrAce Policy Evaluation (GRAPE). It leverages two recent ideas: (1) gap-increasing value update operators in advantage learning for noise-tolerance and (2) off-policy eligibility trace in Retrace algorithm for efficient learning. We provide detailed theoretical analysis of the new algorithm that shows its efficiency and noise-tolerance inherited from Retrace and advantage learning. Furthermore, our analysis shows that GRAPE's learning is significantly efficient than that of a simple learning-rate-based approach while keeping the same level of noise-tolerance. We applied GRAPE to control problems and obtained experimental results supporting our theoretical analysis.