# Explanation-Aware Experience Replay in Rule-Dense Environments

arXiv: https://arxiv.org/abs/2109.14711 · HTML: https://mlsanity.com/paper/2109.14711/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 39 references that are in the map.
- First posted on arXiv 29 Sep 2021.

## Builds on (most cited first, 16 of 16)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2002.00444/) (2020)
- [Model-Based Reinforcement Learning for Atari](https://mlsanity.com/paper/1903.00374/) (2019)
- [Safe Model-based Reinforcement Learning with Stability Guarantees](https://mlsanity.com/paper/1705.08551/) (2017)
- [A Lyapunov-based Approach to Safe Reinforcement Learning](https://mlsanity.com/paper/1805.07708/) (2018)
- [RLlib: Abstractions for Distributed Reinforcement Learning](https://mlsanity.com/paper/1712.09381/) (2017)
- [Experience Replay Optimization](https://mlsanity.com/paper/1906.08387/) (2019)
- [Urban Driving with Multi-Objective Deep Reinforcement Learning](https://mlsanity.com/paper/1811.08586/) (2018)
- [From Philosophy to Interfaces: an Explanatory Method and a Tool Inspired by Achinstein's Theory of Explanation](https://mlsanity.com/paper/2109.04171/) (2021)
- [Culture-Based Explainable Human-Agent Deconfliction](https://mlsanity.com/paper/1911.10098/) (2019)
- [Revisiting Prioritized Experience Replay: A Value Perspective](https://mlsanity.com/paper/2102.03261/) (2021)
- [Combining Experience Replay with Exploration by Random Network Distillation](https://mlsanity.com/paper/1905.07579/) (2019)
- [Agree to Disagree: Subjective Fairness in Privacy-Restricted Decentralised Conflict Resolution](https://mlsanity.com/paper/2107.00032/) (2021)

## Built on by (most cited first, 2 of 2)

- [Attention Loss Adjusted Prioritized Experience Replay](https://mlsanity.com/paper/2309.06684/) (2023)
- [Generating User-Centred Explanations via Illocutionary Question Answering: From Philosophy to Interfaces](https://mlsanity.com/paper/2110.00762/) (2021)

## Abstract (arXiv)

Human environments are often regulated by explicit and complex rulesets. Integrating Reinforcement Learning (RL) agents into such environments motivates the development of learning mechanisms that perform well in rule-dense and exception-ridden environments such as autonomous driving on regulated roads. In this paper, we propose a method for organising experience by means of partitioning the experience buffer into clusters labelled on a per-explanation basis. We present discrete and continuous navigation environments compatible with modular rulesets and 9 learning tasks. For environments with explainable rulesets, we convert rule-based explanations into case-based explanations by allocating state-transitions into clusters labelled with explanations. This allows us to sample experiences in a curricular and task-oriented manner, focusing on the rarity, importance, and meaning of events. We label this concept Explanation-Awareness (XA). We perform XA experience replay (XAER) with intra and inter-cluster prioritisation, and introduce XA-compatible versions of DQN, TD3, and SAC. Performance is consistently superior with XA versions of those algorithms, compared to traditional Prioritised Experience Replay baselines, indicating that explanation engineering can be used in lieu of reward engineering for environments with explainable features.