Toward Interpretable Deep Reinforcement Learning with Linear Model U-Trees
Not on the map. Why?
Abstract (arXiv)
Deep Reinforcement Learning (DRL) has achieved impressive success in many applications. A key component of many DRL models is a neural network representing a Q function, to estimate the expected cumulative reward following a state-action pair. The Q function neural network contains a lot of implicit knowledge about the RL problems, but often remains unexamined and uninterpreted. To our knowledge, this work develops the first mimic learning framework for Q functions in DRL. We introduce Linear Model U-trees (LMUTs) to approximate neural network predictions. An LMUT is learned using a novel on-line algorithm that is well-suited for an active play setting, where the mimic learner observes an ongoing interaction between the neural net and the environment. Empirical evaluation shows that an LMUT mimics a Q function substantially better than five baseline methods. The transparent tree structure of an LMUT facilitates understanding the network's learned knowledge by analyzing feature influence, extracting rules, and highlighting the super-pixels in image inputs.
Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Jul 2018.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 9; 2022: 19; 2023: 15; 2024: 6; 2025: 10; 2026: 6.
65 papers.
Sort:
By area:
- 2021Interpretable Machine Learning: Fundamental Principles and 10 Grand Challengescited by 364
- 2021A Survey on Interpretable Reinforcement Learningcited by 70
- 2023Explainable Deep Reinforcement Learning: State of the Art and Challengescited by 33
- 2022A Survey of Explainable Reinforcement Learningcited by 31
- 2023Concept Learning for Interpretable Multi-Agent Reinforcement Learningcited by 18
- 2021Inherently Explainable Reinforcement Learning in Natural Languagecited by 16
- 2023Deep Explainable Relational Reinforcement Learning: A Neuro-Symbolic Approachcited by 15
- 2022Survey on Fair Reinforcement Learning: Theory and Practicecited by 14
- 2023Explaining Reinforcement Learning Agents Through Counterfactual Action Outcomescited by 13
- 2022Redefining Counterfactual Explanations for Reinforcement Learning: Overview, Challenges and Opportunitiescited by 12
- 2022Explainability in reinforcement learning: perspective and positioncited by 11
- 2021Discovering User-Interpretable Capabilities of Black-Box Planning Agentscited by 10
38 more
- 2022Integrating Policy Summaries with Reward Decomposition for Explaining Reinforcement Learning Agentscited by 10
- 2023Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examplescited by 9
- 2022Sibyl: Adaptive and Extensible Data Placement in Hybrid Storage Systems Using Online Reinforcement Learningcited by 8
- 2022Lazy-MDPs: Towards Interpretable Reinforcement Learning by Learning When to Actcited by 7
- 2021Feature-Based Interpretable Reinforcement Learning based on State-Transition Modelscited by 6
- 2022Explainability in Deep Reinforcement Learning, a Review into Current Methods and Applicationscited by 6
- 2022Constrained Prescriptive Trees via Column Generationcited by 6
- 2024Vanilla Gradient Descent for Oblique Decision Treescited by 6
- 2025Translate Policy to Language: Flow Matching Generated Rewards for LLM Explanationscited by 6
- 2021Neural-to-Tree Policy Distillation with Policy Improvement Criterioncited by 5
- 2022Causal Counterfactuals for Improving the Robustness of Reinforcement Learningcited by 5
- 2023Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challengescited by 5
- 2024Mitigating Information Loss in Tree-Based Reinforcement Learning via Direct Optimizationcited by 5
- 2022There is no Accuracy-Interpretability Tradeoff in Reinforcement Learning for Mazescited by 4
- 2022ProtoX: Explaining a Reinforcement Learning Agent via Prototypingcited by 4
- 2025Model-Agnostic Policy Explanations with Large Language Modelscited by 4
- 2022Policy Regularization for Legible Behaviorcited by 3
- 2022A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challengescited by 3
- 2023Explaining Agent Behavior with Large Language Modelscited by 3
- 2023Understanding Your Agent: Leveraging Large Language Models for Behavior Explanationcited by 3
- 2024SkillTree: Explainable Skill-Based Deep Reinforcement Learning for Long-Horizon Control Taskscited by 3
- 2025Embodied Navigation with Auxiliary Task of Action Description Predictioncited by 3
- 2021Tree-based Focused Web Crawling with Reinforcement Learningcited by 2
- 2022Policy Optimization with Sparse Global Contrastive Explanationscited by 2
- 2022Reward Learning with Trees: Methods and Evaluationcited by 2
- 2024RGMDT: Return-Gap-Minimizing Decision Tree Extraction in Non-Euclidean Metric Spacecited by 2
- 2025Interactive Explanations for Reinforcement-Learning Agentscited by 2
- 2025TalkToAgent: A Human-centric Explanation of Reinforcement Learning Agents with Large Language Modelscited by 2
- 2021Automatic Discovery and Description of Human Planning Strategiescited by 1
- 2022Learning Two-Step Hybrid Policy for Graph-Based Interpretable Reinforcement Learningcited by 1
- 2023XDQN: Inherently Interpretable DQN through Mimickingcited by 1
- 2024Which Experiences Are Influential for RL Agents? Efficiently Estimating The Influence of Experiencescited by 1
- 2025Mxplainer: Explain and Learn Insights by Imitating Mahjong Agentscited by 1
- 2025"So, Tell Me About Your Policy...": Distillation of interpretable policies from Deep Reinforcement Learning agentscited by 1
- 2021Interpretable Local Tree Surrogate Policiescited by 0
- 2022Keeping Minimal Experience to Achieve Efficient Interpretable Policy Distillationcited by 0
- 2022Interpretable Deep Reinforcement Learning for Green Security Games with Real-Time Informationcited by 0
- 2023ASQ-IT: Interactive Explanations for Reinforcement-Learning Agentscited by 0
50 most cited of 65 All 65 (JSON)
- 2026From Black Box to Executable Logic: Explainable Reinforcement Learning through Prolog Expert Systems16 Jul 2026
- 2026Formal Verification of Learned Multi-Agent Communication Policies via Decision Tree Distillation17 Jun 2026
- 2026Critic-Driven Voronoi-Quantization for Distilling Deep RL Policies to Explainable Models14 May 2026
- 2026Approximation-Free Differentiable Oblique Decision Trees8 May 2026
- 2026Principal Prototype Analysis on Manifold for Interpretable Reinforcement Learning30 Mar 2026
- 2026Distilling Deep Reinforcement Learning into Interpretable Fuzzy Rules: An Explainable AI Framework24 Feb 2026
- 2025Embodied Navigation with Auxiliary Task of Action Description Prediction21 Oct 2025
- 2025Self-Explaining Reinforcement Learning for Mobile Network Resource Allocation18 Sep 2025
- 2025TalkToAgent: A Human-centric Explanation of Reinforcement Learning Agents with Large Language Models5 Sep 2025
- 2025"So, Tell Me About Your Policy...": Distillation of interpretable policies from Deep Reinforcement Learning agents10 Jul 2025
- 2025Mxplainer: Explain and Learn Insights by Imitating Mahjong Agents17 Jun 2025
- 2025Strategically Linked Decisions in Long-Term Planning and Reinforcement Learning22 May 2025
38 more
- 2025Model-Agnostic Policy Explanations with Large Language Models8 Apr 2025
- 2025Interactive Explanations for Reinforcement-Learning Agents7 Apr 2025
- 2025Translate Policy to Language: Flow Matching Generated Rewards for LLM Explanations18 Feb 2025
- 2025Compositional Concept-Based Neuron-Level Interpretability for Deep Reinforcement Learning2 Feb 2025
- 2024SkillTree: Explainable Skill-Based Deep Reinforcement Learning for Long-Horizon Control Tasks19 Nov 2024
- 2024RGMDT: Return-Gap-Minimizing Decision Tree Extraction in Non-Euclidean Metric Space21 Oct 2024
- 2024Vanilla Gradient Descent for Oblique Decision Trees17 Aug 2024
- 2024Mitigating Information Loss in Tree-Based Reinforcement Learning via Direct Optimization16 Aug 2024
- 2024Revealing the Learning Process in Reinforcement Learning Agents Through Attention-Oriented Metrics20 Jun 2024
- 2024Which Experiences Are Influential for RL Agents? Efficiently Estimating The Influence of Experiences23 May 2024
- 2023Explaining Reinforcement Learning Agents Through Counterfactual Action Outcomes18 Dec 2023
- 2023Understanding Your Agent: Leveraging Large Language Models for Behavior Explanation29 Nov 2023
- 2023Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examples11 Oct 2023
- 2023Explaining Agent Behavior with Large Language Models19 Sep 2023
- 2023Fidelity-Induced Interpretable Policy Extraction for Reinforcement Learning12 Sep 2023
- 2023Worrisome Properties of Neural Network Controllers and Their Symbolic Representations28 Jul 2023
- 2023Learning Interpretable Models of Aircraft Handling Behaviour by Reinforcement Learning from Human Feedback26 May 2023
- 2023Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challenges17 May 2023
- 2023End-to-End Policy Gradient Method for POMDPs and Explainable Agents19 Apr 2023
- 2023Deep Explainable Relational Reinforcement Learning: A Neuro-Symbolic Approach17 Apr 2023
- 2023Concept Learning for Interpretable Multi-Agent Reinforcement Learning23 Feb 2023
- 2023Which Experiences Are Influential for Your Agent? Policy Iteration with Turn-over Dropout26 Jan 2023
- 2023ASQ-IT: Interactive Explanations for Reinforcement-Learning Agents24 Jan 2023
- 2023Explainable Deep Reinforcement Learning: State of the Art and Challenges24 Jan 2023
- 2023XDQN: Inherently Interpretable DQN through Mimicking8 Jan 2023
- 2022A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challenges12 Nov 2022
- 2022Interpretable Deep Reinforcement Learning for Green Security Games with Real-Time Information9 Nov 2022
- 2022ProtoX: Explaining a Reinforcement Learning Agent via Prototyping6 Nov 2022
- 2022Causal Counterfactuals for Improving the Robustness of Reinforcement Learning2 Nov 2022
- 2022Redefining Counterfactual Explanations for Reinforcement Learning: Overview, Challenges and Opportunities21 Oct 2022
- 2022Integrating Policy Summaries with Reward Decomposition for Explaining Reinforcement Learning Agents21 Oct 2022
- 2022Reward Learning with Trees: Methods and Evaluation3 Oct 2022
- 2022Constrained Prescriptive Trees via Column Generation20 Jul 2022
- 2022Policy Optimization with Sparse Global Contrastive Explanations13 Jul 2022
- 2022Explainability in Deep Reinforcement Learning, a Review into Current Methods and Applications5 Jul 2022
- 2022There is no Accuracy-Interpretability Tradeoff in Reinforcement Learning for Mazes9 Jun 2022
- 2022Survey on Fair Reinforcement Learning: Theory and Practice20 May 2022
- 2022Sibyl: Adaptive and Extensible Data Placement in Hybrid Storage Systems Using Online Reinforcement Learning15 May 2022
50 newest of 65 All 65 (JSON)
- 2025Translate Policy to Language: Flow Matching Generated Rewards for LLM Explanationscited by 6
- 2022Reward Learning with Trees: Methods and Evaluationcited by 2
- 2023Learning Interpretable Models of Aircraft Handling Behaviour by Reinforcement Learning from Human Feedbackcited by 0
- 2026Principal Prototype Analysis on Manifold for Interpretable Reinforcement Learningcited by 0
- 2026Principal Prototype Analysis on Manifold for Interpretable Reinforcement Learning30 Mar 2026
- 2025Translate Policy to Language: Flow Matching Generated Rewards for LLM Explanations18 Feb 2025
- 2023Learning Interpretable Models of Aircraft Handling Behaviour by Reinforcement Learning from Human Feedback26 May 2023
- 2022Reward Learning with Trees: Methods and Evaluation3 Oct 2022
- 2021Interpretable Machine Learning: Fundamental Principles and 10 Grand Challengescited by 364
- 2021A Survey on Interpretable Reinforcement Learningcited by 70
- 2023Explainable Deep Reinforcement Learning: State of the Art and Challengescited by 33
- 2022A Survey of Explainable Reinforcement Learningcited by 31
- 2023Concept Learning for Interpretable Multi-Agent Reinforcement Learningcited by 18
- 2021Inherently Explainable Reinforcement Learning in Natural Languagecited by 16
- 2023Deep Explainable Relational Reinforcement Learning: A Neuro-Symbolic Approachcited by 15
- 2022Survey on Fair Reinforcement Learning: Theory and Practicecited by 14
- 2023Explaining Reinforcement Learning Agents Through Counterfactual Action Outcomescited by 13
- 2022Redefining Counterfactual Explanations for Reinforcement Learning: Overview, Challenges and Opportunitiescited by 12
- 2022Explainability in reinforcement learning: perspective and positioncited by 11
- 2021Discovering User-Interpretable Capabilities of Black-Box Planning Agentscited by 10
12 most cited of 57 All 57 (JSON)
- 2026From Black Box to Executable Logic: Explainable Reinforcement Learning through Prolog Expert Systems16 Jul 2026
- 2026Formal Verification of Learned Multi-Agent Communication Policies via Decision Tree Distillation17 Jun 2026
- 2026Critic-Driven Voronoi-Quantization for Distilling Deep RL Policies to Explainable Models14 May 2026
- 2026Approximation-Free Differentiable Oblique Decision Trees8 May 2026
- 2026Distilling Deep Reinforcement Learning into Interpretable Fuzzy Rules: An Explainable AI Framework24 Feb 2026
- 2025Embodied Navigation with Auxiliary Task of Action Description Prediction21 Oct 2025
- 2025Self-Explaining Reinforcement Learning for Mobile Network Resource Allocation18 Sep 2025
- 2025"So, Tell Me About Your Policy...": Distillation of interpretable policies from Deep Reinforcement Learning agents10 Jul 2025
- 2025Mxplainer: Explain and Learn Insights by Imitating Mahjong Agents17 Jun 2025
- 2025Strategically Linked Decisions in Long-Term Planning and Reinforcement Learning22 May 2025
- 2025Model-Agnostic Policy Explanations with Large Language Models8 Apr 2025
- 2025Interactive Explanations for Reinforcement-Learning Agents7 Apr 2025
12 newest of 57 All 57 (JSON)