mlsanityData to 14 Oct 2026

Generalized Linear Markov Decision Process

arXiv 2506.00818 ↗First posted 1 Jun 2025stat.ML cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

also in 1 more
Abstract (arXiv)

Offline reinforcement learning for longitudinal studies often faces two linked challenges: rewards may be binary or bounded, and reward observations may be available only for a subset of trajectories or time points even when the corresponding state-action-next-state histories are available. Linear Markov decision process methods are tractable because Bellman backups remain linear, but they require linear rewards and do not indicate how transition-only observations should be used. We introduce GRASP-MDP, Generalized Reward And Semi-supervised Pessimism for Markov Decision Processes, a reward-transition separated framework addressing both issues through one Bellman decomposition. It preserves linear transition dynamics while modeling reward means through generalized linear models. Although this breaks the usual linear Bellman form, the backup remains explicit as a nonlinear reward component plus a linear continuation component, yielding a nonlinear-plus-linear Bellman-complete class. In the resulting recursion, observed rewards estimate the generalized reward component, while all available transitions estimate the continuation component. The resulting pessimistic value iteration controls the two estimation errors separately without reward imputation. Finite-sample guarantees show that transition-only observations reduce transition-estimation error while reward uncertainty remains governed by observed rewards. Simulations and a multiple sclerosis electronic health record application illustrate the empirical benefit of retaining transition-only observations.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 29 of its 39 references that are in the map. First posted on arXiv 1 Jun 2025.

Builds on

29 of 39 references.

Closest ancestors · 15

Same direction, newest first

  1. 2023The Provable Benefits of Unsupervised Data Sharing for Offline Reinforcement Learning27 Feb
  2. 2022Semi-Supervised Offline Reinforcement Learning with Action-Free Trajectories12 Oct
  3. 2022Learning Bellman Complete Representations for Offline Policy Evaluation12 Jul
  4. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  5. 2022Nearly Minimax Optimal Offline Reinforcement Learning with Linear Function Approximation: Single-Agent MDP and Markov Game31 May
  6. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning19 Aug
  7. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  8. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  9. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  10. 2020Semi-supervised reward learning for offline reinforcement learning12 Dec
  11. 2020Batch Value-function Approximation with Only Realizability11 Aug
  12. 2020Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluation23 Jun
3 more
  1. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 14

  1. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  2. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  3. 2018Learning to Drive in a Daycited by 273
  4. 2019Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Boundcited by 201
  5. 2018Top-K Off-Policy Correction for a REINFORCE Recommender Systemcited by 182
  6. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  7. 2019Optimism in Reinforcement Learning with Generalized Linear Function Approximationcited by 111
  8. 2018Evaluating Reinforcement Learning Algorithms in Observational Health Settingscited by 60
  9. 2018Finite-sample analysis of M-estimators using self-concordancecited by 30
  10. 2021On the Power of Multitask Representation Learning in Linear MDPcited by 30
  11. 2024A Deployed Online Reinforcement Learning Algorithm In An Oral Health Clinical Trialcited by 11
  12. 2024Harnessing Causality in Reinforcement Learning With Bagged Decision Timescited by 5
2 more
  1. 2019No-regret Exploration in Contextual Reinforcement Learningcited by 4
  2. 2025Reinforcement Learning on Dyads to Enhance Medication Adherencecited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Model-Based Reinforcement Learning Under Confounding
  2. 2025Semi-pessimistic Reinforcement Learning
  3. 2024An MRP Formulation for Supervised Learning: Generalized Temporal Difference Learning Models
  4. 2023Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learning
  5. 2024Reinforcement Learning for Infinite-Horizon Average-Reward Linear MDPs via Approximation by Discounted-Reward MDPs
  6. 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief
  7. 2024Horizon-Free Regret for Linear Markov Decision Processes
  8. 2026A Harmonic Mean Formulation of Average Reward Reinforcement Learning in SMDPs