mlsanityData to 7 Oct 2026

Explainable Reinforcement Learning for Broad-XAI: A Conceptual Framework and Survey

arXiv 2108.09003 ↗First posted 20 Aug 2021cs.AIcited by 24

Not on the map. Why?

Abstract (arXiv)

Broad Explainable Artificial Intelligence moves away from interpreting individual decisions based on a single datum and aims to provide integrated explanations from multiple machine learning algorithms into a coherent explanation of an agent's behaviour that is aligned to the communication needs of the explainee. Reinforcement Learning (RL) methods, we propose, provide a potential backbone for the cognitive model required for the development of Broad-XAI. RL represents a suite of approaches that have had increasing success in solving a range of sequential decision-making problems. However, these algorithms all operate as black-box problem solvers, where they obfuscate their decision-making policy through a complex array of values and functions. EXplainable RL (XRL) is relatively recent field of research that aims to develop techniques to extract concepts from the agent's: perception of the environment; intrinsic/extrinsic motivations/beliefs; Q-values, goals and objectives. This paper aims to introduce a conceptual framework, called the Causal XRL Framework (CXF), that unifies the current XRL research and uses RL as a backbone to the development of Broad-XAI. Additionally, we recognise that RL methods have the ability to incorporate a range of technologies to allow agents to adapt to their environment. CXF is designed for the incorporation of many standard RL extensions and integrated with external ontologies and communication facilities so that the agent can answer questions that explain outcomes and justify its decisions.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 82 of its 277 references that are in the map. First posted on arXiv 20 Aug 2021.

Builds on

82 of 277 references.

References · 82

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  3. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  4. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,201
  5. 2015Prioritized Experience Replaycited by 1,070
  6. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  7. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  8. 2018Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networkscited by 687
  9. 2018Explaining Explanations: An Overview of Interpretability of Machine Learningcited by 508
  10. 2016The Option-Critic Architecturecited by 433
  11. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  12. 2016European Union regulations on algorithmic decision-making and a "right to explanation"cited by 343
38 more
  1. 2017Deep Reinforcement Learning: An Overviewcited by 327
  2. 2019Counterfactual Visual Explanationscited by 312
  3. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 282
  4. 2017StarCraft II: A New Challenge for Reinforcement Learningcited by 268
  5. 2018Explanations based on the Missing: Towards Contrastive Explanations with Pertinent Negativescited by 261
  6. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
  7. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
  8. 2021A Practical Guide to Multi-Objective Reinforcement Learning and Planningcited by 240
  9. 2017Distral: Robust Multitask Reinforcement Learningcited by 231
  10. 2014A Survey of Multi-Objective Sequential Decision-Makingcited by 229
  11. 2018A Comprehensive Survey of Deep Learning for Image Captioningcited by 213
  12. 2020A Review on Generative Adversarial Networks: Algorithms, Theory, and Applicationscited by 209
  13. 2016Learning to Navigate in Complex Environmentscited by 192
  14. 2018Programmatically Interpretable Reinforcement Learningcited by 192
  15. 2019Explainable Reinforcement Learning Through a Causal Lenscited by 177
  16. 2018Deep Reinforcement Learning and the Deadly Triadcited by 170
  17. 2019Multi-Agent Deep Reinforcement Learning for Large-scale Traffic Signal Controlcited by 159
  18. 2018Visual Interpretability for Deep Learning: a Surveycited by 155
  19. 2018Explaining Image Classifiers by Counterfactual Generationcited by 144
  20. 2017Visualizing and Understanding Atari Agentscited by 141
  21. 2020Explainable Reinforcement Learning: A Surveycited by 139
  22. 2017Socially Aware Motion Planning with Deep Reinforcement Learningcited by 128
  23. 2018Model-Based Reinforcement Learning via Meta-Policy Optimizationcited by 99
  24. 2019CXPlain: Causal Explanations for Model Interpretation under Uncertaintycited by 98
  25. 2020Explainability in Deep Reinforcement Learningcited by 96
  26. 2017Imitating Driver Behavior with Generative Adversarial Networkscited by 92
  27. 2017Recent Advances in Recurrent Neural Networkscited by 89
  28. 2017Explainable AI: Beware of Inmates Running the Asylum Or: How I Learnt to Stop Worrying and Love the Social and Behavioural Sciencescited by 84
  29. 2019Automated Rationale Generation: A Technique for Explainable AI and its Effects on Human Perceptionscited by 81
  30. 2016Graying the black box: Understanding DQNscited by 68
  31. 2018Contrastive Explanation: A Structural-Model Approachcited by 64
  32. 2020One Explanation Does Not Fit All: The Promise of Interactive Explanations for Machine Learning Transparencycited by 64
  33. 2017Interpretable & Explorable Approximations of Black Box Modelscited by 54
  34. 2019Interestingness Elements for Explainable Reinforcement Learning: Understanding Agents' Capabilities and Limitationscited by 50
  35. 2018Temporal Difference Variational Auto-Encodercited by 48
  36. 2017Interpretable Policies for Reinforcement Learning by Genetic Programmingcited by 47
  37. 2018Establishing Appropriate Trust via Critical Statescited by 47
  38. 2018Contrastive Explanations for Reinforcement Learning in terms of Expected Consequencescited by 44

50 most cited of 82 All 82 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 8; 2023: 5; 2024: 5; 2025: 3; 2026: 2.

24 papers.

Sort:

By area:

  1. 2021Scalar reward is not enough: A response to Silver, Singh, Precup and Sutton (2021)cited by 55
  2. 2024Interpretable Concept Bottlenecks to Align Reinforcement Learning Agentscited by 23
  3. 2023Automated Cyber Defence: A Reviewcited by 19
  4. 2024End-to-End Neuro-Symbolic Reinforcement Learning with Textual Explanationscited by 17
  5. 2022Redefining Counterfactual Explanations for Reinforcement Learning: Overview, Challenges and Opportunitiescited by 12
  6. 2022Explainability in reinforcement learning: perspective and positioncited by 11
  7. 2022Evaluating Human-like Explanations for Robot Actions in Reinforcement Learning Scenarioscited by 5
  8. 2022Causal Counterfactuals for Improving the Robustness of Reinforcement Learningcited by 5
  9. 2023Efficiently Quantifying Individual Agent Importance in Cooperative MARLcited by 4
  10. 2024Interpretable end-to-end Neurosymbolic Reinforcement Learning agentscited by 4
  11. 2022Boosting human decision-making with AI-generated decision aidscited by 3
  12. 2022A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challengescited by 3
12 more
  1. 2023Advantage Actor-Critic with Reasoner: Explaining the Agent's Behavior from an Exploratory Perspectivecited by 3
  2. 2025Integrating Counterfactual Simulations with Language Models for Explaining Multi-Agent Behaviourcited by 3
  3. 2023On Diagnostics for Understanding Agent Training Behaviour in Cooperative MARLcited by 2
  4. 2025Interactive Explanations for Reinforcement-Learning Agentscited by 2
  5. 2024Issues with Value-Based Multi-objective Reinforcement Learning: Value Function Interference and Overestimation Sensitivitycited by 1
  6. 2025Stackelberg Coupling of Online Representation Learning and Reinforcement Learningcited by 1
  7. 2026SymbXRL: Symbolic Explainable Deep Reinforcement Learning for Mobile Networkscited by 1
  8. 2022Introspection-based Explainable Reinforcement Learning in Episodic and Non-episodic Scenarioscited by 0
  9. 2022Explaining Agent's Decision-making in a Hierarchical Reinforcement Learning Scenariocited by 0
  10. 2023ASQ-IT: Interactive Explanations for Reinforcement-Learning Agentscited by 0
  11. 2024OCALM: Object-Centric Assessment with Language Modelscited by 0
  12. 2026Command-Space Counterfactual Explanations for Pareto-Conditioned Reinforcement Learningcited by 0