mlsanityData to 14 Oct 2026

Solving Offline Reinforcement Learning with Decision Tree Regression

arXiv 2401.11630 ↗First posted 21 Jan 2024cs.LG cs.SY eess.SYcited by 2

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

This study presents a novel approach to addressing offline reinforcement learning (RL) problems by reframing them as regression tasks that can be effectively solved using Decision Trees. Mainly, we introduce two distinct frameworks: return-conditioned and return-weighted decision tree policies (RCDTP and RWDTP), both of which achieve notable speed in agent training as well as inference, with training typically lasting less than a few minutes. Despite the simplification inherent in this reformulated approach to offline RL, our agents demonstrate performance that is at least on par with the established methods. We evaluate our methods on D4RL datasets for locomotion and manipulation, as well as other robotic tasks involving wheeled and flying robots. Additionally, we assess performance in delayed/sparse reward scenarios and highlight the explainability of these policies through action distribution and feature importance.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 35 of its 52 references that are in the map. First posted on arXiv 21 Jan 2024.

Builds on

35 of 52 references.

Closest ancestors · 13

Same direction, newest first

  1. 2022Boosting Offline Reinforcement Learning via Data Rebalancing17 Oct
  2. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
  3. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?12 Apr
  4. 2021d3rlpy: An Offline Deep Reinforcement Learning Library6 Nov
  5. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  6. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  7. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  8. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  9. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
  10. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  11. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  12. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
1 more
  1. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 22

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  8. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  9. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  10. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  11. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  12. 2018Behavioral Cloning from Observationcited by 492
10 more
  1. 2018An Algorithmic Perspective on Imitation Learningcited by 459
  2. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 325
  3. 2023When Do Neural Nets Outperform Boosted Trees on Tabular Data?cited by 223
  4. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 182
  5. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 125
  6. 2019Training Agents using Upside-Down Reinforcement Learningcited by 107
  7. 2019Reward-Conditioned Policiescited by 102
  8. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 91
  9. 2021Learning to Fly -- a Gym Environment with PyBullet Physics for Reinforcement Learning of Multi-agent Quadcopter Controlcited by 77
  10. 2020CDT: Cascading Decision Trees for Explainable Reinforcement Learningcited by 15

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2025Flow-Based Single-Step Completion for Efficient and Expressive Policy Learningcited by 14
  2. 2024F1tenth Autonomous Racing With Offline Reinforcement Learning Methodscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning
  2. 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies
  3. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting
  4. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
  5. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  6. 2020Critic Regularized Regression
  7. 2024Putting the Iterative Training of Decision Trees to the Test on a Real-World Robotic Task
  8. 2024Offline Reinforcement Learning from Datasets with Structured Non-Stationarity