mlsanityData to 14 Oct 2026

Robust Offline Reinforcement Learning with Linearly Structured f-Divergence Regularization

arXiv 2411.18612 ↗First posted 27 Nov 2024cs.LG cs.AI cs.ROcited by 8

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

The Robust Regularized Markov Decision Process (RRMDP) is proposed to learn policies robust to dynamics shifts by adding regularization to the transition dynamics in the value function. Existing methods mostly use unstructured regularization, potentially leading to conservative policies under unrealistic transitions. To address this limitation, we propose a novel framework, the d-rectangular linear RRMDP (d-RRMDP), which introduces latent structures into both transition kernels and regularization. We focus on offline reinforcement learning, where an agent learns policies from a precollected dataset in the nominal environment. We develop the Robust Regularized Pessimistic Value Iteration (R2PVI) algorithm that employs linear function approximation for robust policy learning in d-RRMDPs with f-divergence based regularization terms on transition kernels. We provide instance-dependent upper bounds on the suboptimality gap of R2PVI policies, demonstrating that these bounds are influenced by how well the dataset covers state-action spaces visited by the optimal robust policy under robustly admissible transitions. We establish information-theoretic lower bounds to verify that our algorithm is near-optimal. Finally, numerical experiments validate that R2PVI learns robust policies and exhibits superior computational efficiency compared to baseline methods.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 22 of its 37 references that are in the map. First posted on arXiv 27 Nov 2024.

Builds on

22 of 37 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Sample Complexity of Offline Distributionally Robust Linear Markov Decision Processes19 Mar
  2. 2024Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learning14 Mar
  3. 2023Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coverage27 Oct
  4. 2023Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coverage16 May
  5. 2022Distributionally Robust Offline Reinforcement Learning with Linear Function Approximation14 Sep
  6. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity11 Aug
  7. 2022Nearly Minimax Optimal Offline Reinforcement Learning with Linear Function Approximation: Single-Agent MDP and Markov Game31 May
  8. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 13

  1. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  2. 2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 238
  3. 2018Assessing Generalization in Deep Reinforcement Learningcited by 133
  4. 2022Robust Reinforcement Learning using Offline Datacited by 96
  5. 2021Sample Complexity of Robust Reinforcement Learning with a Generative Modelcited by 83
  6. 2023The Curious Price of Distributional Robustness in Reinforcement Learning with a Generative Modelcited by 52
  7. 2023Improved Sample Complexity Bounds for Distributionally Robust Reinforcement Learningcited by 44
  8. 2024Distributionally Robust Off-Dynamics Reinforcement Learning: Provable Efficiency with Linear Function Approximationcited by 25
  9. 2023Soft Robust MDPs and Risk-Sensitive MDPs: Equivalence, Policy Gradient, and Sample Complexitycited by 17
  10. 2024Off-Dynamics Reinforcement Learning via Domain Adaptation and Reward Augmented Imitationcited by 17
  11. 2024Upper and Lower Bounds for Distributionally Robust Off-Dynamics Reinforcement Learningcited by 16
  12. 2024Return Augmented Decision Transformer for Off-Dynamics Reinforcement Learningcited by 13
1 more
  1. 2023Robust Markov Decision Processes without Model Estimationcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 2.

8 papers.

  1. 2025MOBODY: Model Based Off-Dynamics Offline Reinforcement Learningcited by 9
  2. 2025ORVIT: Near-Optimal Online Distributionally Robust Reinforcement Learningcited by 6
  3. 2025Linear Mixture Distributionally Robust Markov Decision Processescited by 5
  4. 2025Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Dualitycited by 4
  5. 2025Online Robust Reinforcement Learning with General Function Approximationcited by 2
  6. 2026Robust Data-Collection Policy Learning for Low-Variance Online Policy Evaluationcited by 2
  7. 2025Finite-Time Convergence of Single-Trajectory Chi-Square Robust Q-Learning With Linear Function Approximationcited by 0
  8. 2026Cross-Domain Energy-Guided Diffusion Generation for Off-Dynamics Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Robust Regularized Policy Iteration under Transition Uncertainty
  2. 2025Policy Regularized Distributionally Robust Markov Decision Processes with Linear Function Approximation
  3. 2024Model-Free Robust φ-Divergence Reinforcement Learning Using Both Offline and Online Data
  4. 2024Robust Offline Reinforcement Learning for Non-Markovian Decision Processes
  5. 2021Robust Entropy-regularized Markov Decision Processes
  6. 2025State Entropy Regularization for Robust Reinforcement Learning
  7. 2021DROMO: Distributionally Robust Offline Model-based Policy Optimization
  8. 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learning