# Conservative Q-Improvement: Reinforcement Learning for an Interpretable Decision-Tree Policy

arXiv: https://arxiv.org/abs/1907.01180 · HTML: https://mlsanity.com/paper/1907.01180/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jul 2019.

## Built on by (most cited first, 29 of 29)

- [Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges](https://mlsanity.com/paper/2103.11251/) (2021)
- [A Survey on Interpretable Reinforcement Learning](https://mlsanity.com/paper/2112.13112/) (2021)
- [Bridging the Human-AI Knowledge Gap: Concept Discovery and Transfer in AlphaZero](https://mlsanity.com/paper/2310.16410/) (2023)
- [Explainable Deep Reinforcement Learning: State of the Art and Challenges](https://mlsanity.com/paper/2301.09937/) (2023)
- [Iterative Bounding MDPs: Learning Interpretable Policies via Non-Interpretable Methods](https://mlsanity.com/paper/2102.13045/) (2021)
- [MAVIPER: Learning Decision Tree Policies for Interpretable Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2205.12449/) (2022)
- [Explainability in reinforcement learning: perspective and position](https://mlsanity.com/paper/2203.11547/) (2022)
- [MIXRTs: Toward Interpretable Multi-Agent Reinforcement Learning via Mixing Recurrent Soft Decision Trees](https://mlsanity.com/paper/2209.07225/) (2022)
- [Feature-Based Interpretable Reinforcement Learning based on State-Transition Models](https://mlsanity.com/paper/2105.07099/) (2021)
- [Vanilla Gradient Descent for Oblique Decision Trees](https://mlsanity.com/paper/2408.09135/) (2024)
- [Mitigating Information Loss in Tree-Based Reinforcement Learning via Direct Optimization](https://mlsanity.com/paper/2408.08761/) (2024)
- [There is no Accuracy-Interpretability Tradeoff in Reinforcement Learning for Mazes](https://mlsanity.com/paper/2206.04266/) (2022)
- [Optimizing Interpretable Decision Tree Policies for Reinforcement Learning](https://mlsanity.com/paper/2408.11632/) (2024)
- [XAI-N: Sensor-based Robot Navigation using Expert Policies and Decision Trees](https://mlsanity.com/paper/2104.10818/) (2021)
- [A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challenges](https://mlsanity.com/paper/2211.06665/) (2022)
- [SMOSE: Sparse Mixture of Shallow Experts for Interpretable Reinforcement Learning in Continuous Control Tasks](https://mlsanity.com/paper/2412.13053/) (2024)
- [Information Templates: A New Paradigm for Intelligent Active Feature Acquisition](https://mlsanity.com/paper/2508.18380/) (2025)
- [Embodied Navigation with Auxiliary Task of Action Description Prediction](https://mlsanity.com/paper/2510.21809/) (2025)
- [Reward Learning with Trees: Methods and Evaluation](https://mlsanity.com/paper/2210.01007/) (2022)
- [RGMDT: Return-Gap-Minimizing Decision Tree Extraction in Non-Euclidean Metric Space](https://mlsanity.com/paper/2410.16517/) (2024)
- [(When) Are Contrastive Explanations of Reinforcement Learning Helpful?](https://mlsanity.com/paper/2211.07719/) (2022)
- [XDQN: Inherently Interpretable DQN through Mimicking](https://mlsanity.com/paper/2301.03043/) (2023)
- [How much informative is your XAI? A decision-making assessment task to objectively measure the goodness of explanations](https://mlsanity.com/paper/2312.04379/) (2023)
- [Methodology for Interpretable Reinforcement Learning for Optimizing Mechanical Ventilation](https://mlsanity.com/paper/2404.03105/) (2024)
- [MSVIPER: Improved Policy Distillation for Reinforcement-Learning-Based Robot Navigation](https://mlsanity.com/paper/2209.09079/) (2022)
- [Revealing the Learning Process in Reinforcement Learning Agents Through Attention-Oriented Metrics](https://mlsanity.com/paper/2406.14324/) (2024)
- [Let people fail! Exploring the influence of explainable virtual and robotic agents in learning-by-doing tasks](https://mlsanity.com/paper/2411.10176/) (2024)
- [Approximation-Free Differentiable Oblique Decision Trees](https://mlsanity.com/paper/2605.07837/) (2026)
- [Auditability Is Not One Property: Rule Overlap, Behavioural Agreement, and Composition in Reinforcement Learning](https://mlsanity.com/paper/2609.28581/) (2026)

## Abstract (arXiv)

There is a growing desire in the field of reinforcement learning (and machine learning in general) to move from black-box models toward more "interpretable AI." We improve interpretability of reinforcement learning by increasing the utility of decision tree policies learned via reinforcement learning. These policies consist of a decision tree over the state space, which requires fewer parameters to express than traditional policy representations. Existing methods for creating decision tree policies via reinforcement learning focus on accurately representing an action-value function during training, but this leads to much larger trees than would otherwise be required. To address this shortcoming, we propose a novel algorithm which only increases tree size when the estimated discounted future reward of the overall policy would increase by a sufficient amount. Through evaluation in a simulated environment, we show that its performance is comparable or superior to traditional tree-based approaches and that it yields a more succinct policy. Additionally, we discuss tuning parameters to control the tradeoff between optimizing for smaller tree size or for overall reward.