# Optimizing Interpretable Decision Tree Policies for Reinforcement Learning

arXiv: https://arxiv.org/abs/2408.11632 · HTML: https://mlsanity.com/paper/2408.11632/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 11 of its 27 references that are in the map.
- First posted on arXiv 21 Aug 2024.

## Placed in

- Offline RL and imitation learning from data / [Decision transformers and sequence modeling](https://mlsanity.com/offlinerl/decision-transformers-and-sequence-modeling/)
- Deep reinforcement learning methods / [Interpretable and logic-guided RL](https://mlsanity.com/deeprl/interpretable-and-logic-guided-rl/)

## Builds on (most cited first, 11 of 11)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Verifiable Reinforcement Learning via Policy Extraction](https://mlsanity.com/paper/1805.08328/) (2018)
- [A Survey on Interpretable Reinforcement Learning](https://mlsanity.com/paper/2112.13112/) (2021)
- [Iterative Bounding MDPs: Learning Interpretable Policies via Non-Interpretable Methods](https://mlsanity.com/paper/2102.13045/) (2021)
- [A Survey of Explainable Reinforcement Learning](https://mlsanity.com/paper/2202.08434/) (2022)
- [Conservative Q-Improvement: Reinforcement Learning for an Interpretable Decision-Tree Policy](https://mlsanity.com/paper/1907.01180/) (2019)
- [Learning Interpretable, High-Performing Policies for Autonomous Driving](https://mlsanity.com/paper/2202.02352/) (2022)

## Built on by (most cited first, 4 of 4)

- [High-order Interactions Modeling for Interpretable Multi-Agent Q-Learning](https://mlsanity.com/paper/2510.20218/) (2025)
- [A Multi-Component Reward Function with Policy Gradient for Automated Feature Selection with Dynamic Regularization and Bias Mitigation](https://mlsanity.com/paper/2510.09705/) (2025)
- [Interpretable Policy Distillation for Power Grid Topology Control](https://mlsanity.com/paper/2606.00561/) (2026)
- [Explainable Reinforcement Learning via Physics-Aware Policy Distillation](https://mlsanity.com/paper/2607.24672/) (2026)

## Abstract (arXiv)

Reinforcement learning techniques leveraging deep learning have made tremendous progress in recent years. However, the complexity of neural networks prevents practitioners from understanding their behavior. Decision trees have gained increased attention in supervised learning for their inherent interpretability, enabling modelers to understand the exact prediction process after learning. This paper considers the problem of optimizing interpretable decision tree policies to replace neural networks in reinforcement learning settings. Previous works have relaxed the tree structure, restricted to optimizing only tree leaves, or applied imitation learning techniques to approximately copy the behavior of a neural network policy with a decision tree. We propose the Decision Tree Policy Optimization (DTPO) algorithm that directly optimizes the complete decision tree using policy gradients. Our technique uses established decision tree heuristics for regression to perform policy optimization. We empirically show that DTPO is a competitive algorithm compared to imitation learning algorithms for optimizing decision tree policies in reinforcement learning.