mlsanityData to 7 Oct 2026

A Survey on Interpretable Reinforcement Learning

arXiv 2112.13112 ↗First posted 24 Dec 2021cs.LG cs.AIcited by 70

Not on the map. Why?

Abstract (arXiv)

Although deep reinforcement learning has become a promising machine learning approach for sequential decision-making problems, it is still not mature enough for high-stake domains such as autonomous driving or medical applications. In such contexts, a learned policy needs for instance to be interpretable, so that it can be inspected before any deployment (e.g., for safety and verifiability reasons). This survey provides an overview of various approaches to achieve higher interpretability in reinforcement learning (RL). To that aim, we distinguish interpretability (as a property of a model) and explainability (as a post-hoc operation, with the intervention of a proxy) and discuss them in the context of RL with an emphasis on the former notion. In particular, we argue that interpretable RL may embrace different facets: interpretable inputs, interpretable (transition/reward) models, and interpretable decision-making. Based on this scheme, we summarize and analyze recent work related to interpretable RL with an emphasis on papers published in the past 10 years. We also discuss briefly some related research areas and point to some potential promising research directions.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Builds on 133 of its 292 references that are in the map. First posted on arXiv 24 Dec 2021.

Builds on

133 of 292 references.

References · 133

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  5. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  6. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  7. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  10. 2017Neural Message Passing for Quantum Chemistrycited by 3,460
  11. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  12. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,911
38 more
  1. 2016Concrete Problems in AI Safetycited by 1,582
  2. 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,543
  3. 2018Relational inductive biases, deep learning, and graph networkscited by 1,341
  4. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,201
  5. 2019Attention is not Explanationcited by 916
  6. 2017Simple Online and Realtime Tracking with a Deep Association Metriccited by 908
  7. 2015Gated Graph Sequence Neural Networkscited by 792
  8. 2017Deep Reinforcement Learning that Matterscited by 727
  9. 2016The Mythos of Model Interpretabilitycited by 716
  10. 2019Solving Rubik's Cube with a Robot Handcited by 699
  11. 2019Attention is not not Explanationcited by 629
  12. 2018Explaining Explanations: An Overview of Interpretability of Machine Learningcited by 508
  13. 2016Interaction Networks for Learning about Objects, Relations and Physicscited by 503
  14. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  15. 2018An Algorithmic Perspective on Imitation Learningcited by 455
  16. 2017Safe Reinforcement Learning via Shieldingcited by 427
  17. 2015Policy Distillationcited by 414
  18. 2020Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditingcited by 380
  19. 2017A simple neural network module for relational reasoningcited by 345
  20. 2020Discovering Symbolic Models from Deep Learning with Inductive Biasescited by 335
  21. 2017Adversarial Attacks on Neural Network Policiescited by 314
  22. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 294
  23. 2018Towards a Definition of Disentangled Representationscited by 294
  24. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  25. 2017StarCraft II: A New Challenge for Reinforcement Learningcited by 268
  26. 2017A Semantic Loss Function for Deep Learning with Symbolic Knowledgecited by 267
  27. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 256
  28. 2018Graph networks as learnable physics engines for inference and controlcited by 242
  29. 2016Model-Agnostic Interpretability of Machine Learningcited by 219
  30. 2017Learning Explanatory Rules from Noisy Datacited by 210
  31. 2016Modular Multitask Reinforcement Learning with Policy Sketchescited by 200
  32. 2018Programmatically Interpretable Reinforcement Learningcited by 192
  33. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 188
  34. 2019Explainable Reinforcement Learning Through a Causal Lenscited by 177
  35. 2018Verifiable Reinforcement Learning via Policy Extractioncited by 176
  36. 2018A Study on Overfitting in Deep Reinforcement Learningcited by 170
  37. 2019Neural Logic Machinescited by 144
  38. 2017Visualizing and Understanding Atari Agentscited by 141

50 most cited of 133 All 133 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 16; 2024: 21; 2025: 20; 2026: 7.

70 papers.

Sort:

By area:

  1. 2023Bridging the Human-AI Knowledge Gap: Concept Discovery and Transfer in AlphaZerocited by 41
  2. 2025Mechanistic interpretability for steering vision-language-action modelscited by 30
  3. 2023Learning World Models with Identifiable Factorizationcited by 23
  4. 2024CoMAL: Collaborative Multi-Agent Large Language Models for Mixed-Autonomy Trafficcited by 22
  5. 2023Automated Cyber Defence: A Reviewcited by 19
  6. 2024Interpretable and Editable Programmatic Tree Policies for Reinforcement Learningcited by 18
  7. 2023Semantic HELM: A Human-Readable Memory for Reinforcement Learningcited by 17
  8. 2025General agents contain world modelscited by 14
  9. 2022Explainability in reinforcement learning: perspective and positioncited by 11
  10. 2023Aligning Robot and Human Representationscited by 11
  11. 2023Transformers in Reinforcement Learning: A Surveycited by 11
  12. 2023Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examplescited by 9
38 more
  1. 2023Optimal Decision Tree Policies for Markov Decision Processescited by 8
  2. 2024Interpretable Clustering: A Surveycited by 8
  3. 2025Multimodal LLM-assisted Evolutionary Search for Programmatic Control Policiescited by 8
  4. 2023Interpretable Machine Learning for Discovery: Statistical Challenges & Opportunitiescited by 7
  5. 2024Verco: Learning Coordinated Verbal Communication for Multi-agent Reinforcement Learningcited by 7
  6. 2023Reinforcement Learning with Knowledge Representation and Reasoning: A Brief Surveycited by 5
  7. 2025Simulating the Unseen: Crash Prediction Must Learn from What Did Not Happencited by 5
  8. 2023Interpretable Reinforcement Learning via Neural Additive Models for Inventory Managementcited by 4
  9. 2023Efficiently Quantifying Individual Agent Importance in Cooperative MARLcited by 4
  10. 2024Interpretability in Action: Exploratory Analysis of VPT, a Minecraft Agentcited by 4
  11. 2024LICORICE: Label-Efficient Concept-Based Interpretable Reinforcement Learningcited by 4
  12. 2024Optimizing Interpretable Decision Tree Policies for Reinforcement Learningcited by 4
  13. 2024Interpretable end-to-end Neurosymbolic Reinforcement Learning agentscited by 4
  14. 2025A Survey of Reinforcement Learning for Optimization in Automationcited by 4
  15. 2025From monoliths to modules: Decomposing transducers for efficient world modellingcited by 4
  16. 2022A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challengescited by 3
  17. 2023Leveraging Reward Consistency for Interpretable Feature Discovery in Reinforcement Learningcited by 3
  18. 2024Synthesizing Interpretable Control Policies through Large Language Model Guided Searchcited by 3
  19. 2024SMOSE: Sparse Mixture of Shallow Experts for Interpretable Reinforcement Learning in Continuous Control Taskscited by 3
  20. 2025Evaluating Interpretable Reinforcement Learning by Distilling Policies into Programscited by 3
  21. 2025Demystifying the Mechanisms Behind Emergent Exploration in Goal-conditioned RLcited by 3
  22. 2022Boolean Decision Rules for Reinforcement Learning Policy Summarisationcited by 2
  23. 2022Explainable Reinforcement Learning on Financial Stock Trading using SHAPcited by 2
  24. 2022Reward Learning with Trees: Methods and Evaluationcited by 2
  25. 2022Disentangled (Un)Controllable Featurescited by 2
  26. 2024Model-based Policy Optimization using Symbolic World Modelcited by 2
  27. 2024Empowering Clinicians with Medical Decision Transformers: A Framework for Sepsis Treatmentcited by 2
  28. 2024KnowPC: Knowledge-Driven Programmatic Reinforcement Learning for Zero-shot Coordinationcited by 2
  29. 2024Towards certifiable AI in aviation: landscape, challenges, and opportunitiescited by 2
  30. 2025Information Science Principles of Machine Learning: A Causal Chain Meta-Framework Based on Formalized Information Mappingcited by 2
  31. 2025Sparsity-Driven Plasticity in Multi-Task Reinforcement Learningcited by 2
  32. 2025Information Templates: A New Paradigm for Intelligent Active Feature Acquisitioncited by 2
  33. 2026Interpreting Agentic Systems: Beyond Model Explanations to System-Level Accountabilitycited by 2
  34. 2023Efficient Open-world Reinforcement Learning via Knowledge Distillation and Autonomous Rule Discoverycited by 1
  35. 2023How much can change in a year? Revisiting Evaluation in Multi-Agent Reinforcement Learningcited by 1
  36. 2024An Explainable Deep Reinforcement Learning Model for Warfarin Maintenance Dosing Using Policy Distillation and Action Forgingcited by 1
  37. 2024In Search of Trees: Decision-Tree Policy Synthesis for Black-Box Systems via Searchcited by 1
  38. 2025Inducing, Detecting and Characterising Neural Modules: A Pipeline for Functional Interpretability in Reinforcement Learningcited by 1

50 most cited of 70 All 70 (JSON)