mlsanityData to 7 Oct 2026

Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coverage

arXiv 2310.18434 ↗First posted 27 Oct 2023cs.LG stat.MLcited by 12

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

The goal of an offline reinforcement learning (RL) algorithm is to learn optimal polices using historical (offline) data, without access to the environment for online exploration. One of the main challenges in offline RL is the distribution shift which refers to the difference between the state-action visitation distribution of the data generating policy and the learning policy. Many recent works have used the idea of pessimism for developing offline RL algorithms and characterizing their sample complexity under a relatively weak assumption of single policy concentrability. Different from the offline RL literature, the area of distributionally robust learning (DRL) offers a principled framework that uses a minimax formulation to tackle model mismatch between training and testing environments. In this work, we aim to bridge these two areas by showing that the DRL approach can be used to tackle the distributional shift problem in offline RL. In particular, we propose two offline RL algorithms using the DRL framework, for the tabular and linear function approximation settings, and characterize their sample complexity under the single policy concentrability assumption. We also demonstrate the superior performance our proposed algorithm through simulation experiments.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 51 of its 80 references that are in the map. First posted on arXiv 27 Oct 2023.

Builds on

51 of 80 references.

Closest ancestors · 15

Same direction, newest first

  1. 2022Distributionally Robust Offline Reinforcement Learning with Linear Function Approximation14 Sep
  2. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity11 Aug
  3. 2022Nearly Minimax Optimal Offline Reinforcement Learning with Linear Function Approximation: Single-Agent MDP and Markov Game31 May
  4. 2022Settling the Sample Complexity of Model-Based Offline Reinforcement Learning11 Apr
  5. 2022Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimism11 Mar
  6. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexity28 Feb
  7. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage13 Jul
  8. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  9. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  10. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  11. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  12. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?22 Oct
3 more
  1. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  2. 2020MOPO: Model-based Offline Policy Optimization27 May
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 36

  1. 2016OpenAI Gymcited by 1,684
  2. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  3. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2021Generalized Out-of-Distribution Detection: A Surveycited by 796
  7. 2016CVXPY: A Python-Embedded Modeling Language for Convex Optimizationcited by 633
  8. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  9. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  10. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 452
  11. 2018Learning Models with Uniform Performance via Distributionally Robust Optimizationcited by 334
  12. 2015Data-driven Distributionally Robust Optimization Using the Wasserstein Metric: Performance Guarantees and Tractable Reformulationscited by 285
24 more
  1. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  2. 2020Large-Scale Methods for Distributionally Robust Optimizationcited by 179
  3. 2019Wasserstein Distributionally Robust Optimization: Theory and Applications in Machine Learningcited by 169
  4. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  5. 2021Online Robust Reinforcement Learning with Model Uncertaintycited by 95
  6. 2022Robust Reinforcement Learning using Offline Datacited by 95
  7. 2021Sample Complexity of Robust Reinforcement Learning with a Generative Modelcited by 82
  8. 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
  9. 2022Policy Gradient Method For Robust Reinforcement Learningcited by 67
  10. 2020Distributionally Robust Losses for Latent Covariate Mixturescited by 57
  11. 2017Reinforcement Learning under Model Mismatchcited by 52
  12. 2021Distributionally Robust Learningcited by 44
  13. 2023Improved Sample Complexity Bounds for Distributionally Robust Reinforcement Learningcited by 44
  14. 2018Convergence and Concentration of Empirical Measures under Wasserstein Distance in Unbounded Functional Spacescited by 41
  15. 2022Policy Gradient in Robust MDPs with Global Convergence Guaranteecited by 35
  16. 2016A Minimax Approach to Supervised Learningcited by 33
  17. 2022First-order Policy Optimization for Robust Markov Decision Processcited by 31
  18. 2023Policy Gradient for Rectangular Robust Markov Decision Processescited by 29
  19. 2020Scalable First-Order Methods for Robust MDPscited by 22
  20. 2020Near-Optimal Learning of Tree-Structured Distributions by Chow-Liucited by 20
  21. 2022Efficient Policy Iteration for Robust Markov Decision Processes via Regularizationcited by 19
  22. 2014Empirical Q-Value Iterationcited by 7
  23. 2013Empirical Dynamic Programmingcited by 6
  24. 2023Near-Optimal Degree Testing for Bayes Netscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 5; 2026: 3.

12 papers.

  1. 2024Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learningcited by 17
  2. 2025Robust LLM Alignment via Distributionally Robust Direct Preference Optimizationcited by 17
  3. 2024Robust Offline Reinforcement Learning with Linearly Structured f-Divergence Regularizationcited by 8
  4. 2025Linear Mixture Distributionally Robust Markov Decision Processescited by 5
  5. 2024Tractable Equilibrium Computation in Markov Games through Risk Aversioncited by 3
  6. 2024Model-Free Robust φ-Divergence Reinforcement Learning Using Both Offline and Online Datacited by 2
  7. 2025DGRO: Enhancing LLM Reasoning via Exploration-Exploitation Control and Reward Variance Managementcited by 1
  8. 2025Balance Equation-based Distributionally Robust Offline Imitation Learningcited by 1
  9. 2026Online Robust Reinforcement Learning Through Monte-Carlo Planningcited by 1
  10. 2025DiSA-IQL: Offline Reinforcement Learning for Robust Soft Robot Control under Distribution Shiftscited by 0
  11. 2026Post Hoc Extraction of Pareto Fronts for Continuous Controlcited by 0
  12. 2026When Dynamics Shift, Robust Task Inference Wins: Offline Imitation Learning with Behavior Foundation Models Revisitedcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  2. 2023Achieving the Asymptotically Optimal Sample Complexity of Offline Reinforcement Learning: A DRO-Based Approach
  3. 2025On the Statistical Complexity for Offline and Low-Adaptive Reinforcement Learning with Structures
  4. 2024Robust Offline Reinforcement Learning for Non-Markovian Decision Processes
  5. 2023Selective Uncertainty Propagation in Offline RL
  6. 2022Offline RL Policies Should be Trained to be Adaptive
  7. 2022Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learning
  8. 2021Instabilities of Offline RL with Pre-Trained Neural Representation