# Attention Loss Adjusted Prioritized Experience Replay

arXiv: https://arxiv.org/abs/2309.06684 · HTML: https://mlsanity.com/paper/2309.06684/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 33 references that are in the map.
- First posted on arXiv 13 Sep 2023.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Actor-Attention-Critic for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1810.02912/) (2018)
- [An Equivalence between Loss Functions and Non-Uniform Sampling in Experience Replay](https://mlsanity.com/paper/2007.06049/) (2020)
- [Generalized Huber Loss for Robust Learning and its Efficient Minimization for a Robust Statistics](https://mlsanity.com/paper/2108.12627/) (2021)
- [The Effects of Memory Replay in Reinforcement Learning](https://mlsanity.com/paper/1710.06574/) (2017)
- [Actor Prioritized Experience Replay](https://mlsanity.com/paper/2209.00532/) (2022)
- [Planning by Prioritized Sweeping with Small Backups](https://mlsanity.com/paper/1301.2343/) (2013)
- [Deep Q-Learning with Q-Matrix Transfer Learning for Novel Fire Evacuation Environment](https://mlsanity.com/paper/1905.09673/) (2019)
- [Explanation-Aware Experience Replay in Rule-Dense Environments](https://mlsanity.com/paper/2109.14711/) (2021)

## Built on by (most cited first, 1 of 1)

- [Directly Attention Loss Adjusted Prioritized Experience Replay](https://mlsanity.com/paper/2311.14390/) (2023)

## Abstract (arXiv)

Prioritized Experience Replay (PER) is a technical means of deep reinforcement learning by selecting experience samples with more knowledge quantity to improve the training rate of neural network. However, the non-uniform sampling used in PER inevitably shifts the state-action space distribution and brings the estimation error of Q-value function. In this paper, an Attention Loss Adjusted Prioritized (ALAP) Experience Replay algorithm is proposed, which integrates the improved Self-Attention network with Double-Sampling mechanism to fit the hyperparameter that can regulate the importance sampling weights to eliminate the estimation error caused by PER. In order to verify the effectiveness and generality of the algorithm, the ALAP is tested with value-function based, policy-gradient based and multi-agent reinforcement learning algorithms in OPENAI gym, and comparison studies verify the advantage and efficiency of the proposed training framework.