mlsanityData to 14 Oct 2026

Reinforcement Learning for Individual Optimal Policy from Heterogeneous Data

arXiv 2505.09496 ↗First posted 14 May 2025stat.ML cs.LGcited by 4

On the map: Offline RL and imitation learning from data → Offline RL algorithms

also in 1 more
Abstract (arXiv)

Offline reinforcement learning (RL) aims to find optimal policies in dynamic environments in order to maximize the expected total rewards by leveraging pre-collected data. Learning from heterogeneous data is one of the fundamental challenges in offline RL. Traditional methods focus on learning an optimal policy for all individuals with pre-collected data from a single episode or homogeneous batch episodes, and thus, may result in a suboptimal policy for a heterogeneous population. In this paper, we propose an individualized offline policy optimization framework for heterogeneous time-stationary Markov decision processes (MDPs). The proposed heterogeneous model with individual latent variables enables us to efficiently estimate the individual Q-functions, and our Penalized Pessimistic Personalized Policy Learning (P4L) algorithm guarantees a fast rate on the average regret under a weak partial coverage assumption on behavior policies. In addition, our simulation studies and a real data application demonstrate the superior numerical performance of the proposed method compared with existing methods.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 25 of its 37 references that are in the map. First posted on arXiv 14 May 2025.

Builds on

25 of 37 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrability9 Feb
  2. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  3. 2021PerSim: Data-Efficient Offline Reinforcement Learning with Heterogeneous Agents via Personalized Simulators13 Feb
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 21

  1. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  2. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  3. 2015Estimation and Inference of Heterogeneous Treatment Effects using Random Forestscited by 600
  4. 2017Meta-learners for Estimating Heterogeneous Treatment Effects using Machine Learningcited by 478
  5. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 433
  6. 2017Quasi-Oracle Estimation of Heterogeneous Treatment Effectscited by 332
  7. 2019Batch Policy Learning under Constraintscited by 244
  8. 2023A Tutorial on Meta-Reinforcement Learningcited by 108
  9. 2020Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learningcited by 98
  10. 2020Offline Meta-Reinforcement Learning with Advantage Weightingcited by 93
  11. 2017Deep Reinforcement Learning for Sepsis Treatmentcited by 71
  12. 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
9 more
  1. 2016Estimating Dynamic Treatment Regimes in Mobile Health Using V-learningcited by 65
  2. 2015Residual Weighted Learning for Estimating Individualized Treatment Rulescited by 38
  3. 2021Estimating Optimal Infinite Horizon Dynamic Treatment Regimes via pT-Learningcited by 26
  4. 2021Unifying Cardiovascular Modelling with Deep Reinforcement Learning for Uncertainty Aware Control of Sepsis Treatmentcited by 16
  5. 2021Provably Efficient Multi-Task Reinforcement Learning with Model Transfercited by 16
  6. 2022Reinforcement Learning with Heterogeneous Data: Estimation and Inferencecited by 7
  7. 2017Individualized Multi-directional Variable Selectioncited by 5
  8. 2022Doubly Inhomogeneous Reinforcement Learningcited by 3
  9. 2022Heterogeneous Synthetic Learner for Panel Datacited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026Adaptive Auto-Harness: Sustained Self-Improvement for Agentic System Deployment on Open-Ended Task Streamscited by 27
  2. 2025Generalized Fitted Q-Iteration with Clustered Datacited by 0
  3. 2026Personalization Meets Safety:Mechanisms,Risks,and Mitigations in Personalized LLMscited by 0
  4. 2026Deep Weighted Bellman Residual Minimization for Q^* Estimationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Provably Efficient Offline Reinforcement Learning with Perturbed Data Sources
  2. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning
  3. 2026A Unified Algorithmic Framework for Hybrid Reinforcement Learning in Tabular MDPs with Shifted Transition Dynamics
  4. 2022Nearly Minimax Optimal Offline Reinforcement Learning with Linear Function Approximation: Single-Agent MDP and Markov Game
  5. 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief
  6. 2021Near-Optimal Offline Reinforcement Learning via Double Variance Reduction
  7. 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimism
  8. 2020MOPO: Model-based Offline Policy Optimization