mlsanityData to 14 Oct 2026

Distributionally Robust Offline Reinforcement Learning with Linear Function Approximation

arXiv 2209.06620 ↗First posted 14 Sep 2022cs.LG cs.AI stat.MLcited by 32

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Among the reasons hindering reinforcement learning (RL) applications to real-world problems, two factors are critical: limited data and the mismatch between the testing environment (real environment in which the policy is deployed) and the training environment (e.g., a simulator). This paper attempts to address these issues simultaneously with distributionally robust offline RL, where we learn a distributionally robust policy using historical data obtained from the source environment by optimizing against a worst-case perturbation thereof. In particular, we move beyond tabular settings and consider linear function approximation. More specifically, we consider two settings, one where the dataset is well-explored and the other where the dataset has sufficient coverage of the optimal policy. We propose two algorithms~-- one for each of the two settings~-- that achieve error bounds tildeO(d1/2/N1/2) and tildeO(d3/2/N1/2) respectively, where d is the dimension in the linear function approximation and N is the number of trajectories in the dataset. To the best of our knowledge, they provide the first non-asymptotic results of the sample complexity in this setting. Diverse experiments are conducted to demonstrate our theoretical findings, showing the superiority of our algorithm against the non-robust one.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 55 of its 90 references that are in the map. First posted on arXiv 14 Sep 2022.

Builds on

55 of 90 references.

Closest ancestors · 20

Same direction, newest first

  1. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity11 Aug
  2. 2022Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimism11 Mar
  3. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
  4. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  5. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  6. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning19 Aug
  7. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  8. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  9. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  10. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  11. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  12. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
8 more
  1. 2020Offline Reinforcement Learning from Images with Latent Space Models21 Dec
  2. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
  3. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?22 Oct
  4. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  5. 2020MOPO: Model-based Offline Policy Optimization27 May
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  7. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  8. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 35

  1. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  3. 2016End to End Learning for Self-Driving Carscited by 1,009
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  5. 2017Robust Adversarial Reinforcement Learningcited by 451
  6. 2019A Survey of Autonomous Driving: Common Practices and Emerging Technologiescited by 379
  7. 2018Learning Models with Uniform Performance via Distributionally Robust Optimizationcited by 335
  8. 2017Adversarial Attacks on Neural Network Policiescited by 320
  9. 2019Reinforcement Learning in Healthcare: A Surveycited by 310
  10. 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 292
  11. 2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 238
  12. 2016Statistics of Robust Optimization: A Generalized Empirical Likelihood Approachcited by 183
23 more
  1. 2020Learning Near Optimal Policies with Low Inherent Bellman Errorcited by 181
  2. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 177
  3. 2016Variance-based regularization with convex objectivescited by 155
  4. 2017Robust Deep Reinforcement Learning with Adversarial Attackscited by 154
  5. 2019Is a Good Representation Sufficient for Sample Efficient Reinforcement Learning?cited by 140
  6. 2016EPOpt: Learning Robust Neural Network Policies Using Model Ensemblescited by 138
  7. 2021Provably Efficient Reinforcement Learning with Linear Function Approximation Under Adaptivity Constraintscited by 124
  8. 2019Optimism in Reinforcement Learning with Generalized Linear Function Approximationcited by 111
  9. 2018Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendationcited by 101
  10. 2019Sample Complexity of Reinforcement Learning using Linearly Combined Model Ensemblescited by 100
  11. 2022Robust Reinforcement Learning using Offline Datacited by 96
  12. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 91
  13. 2013Data-Driven Robust Optimizationcited by 85
  14. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 79
  15. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  16. 2020Exponential Lower Bounds for Batch Reinforcement Learning: Batch RL can be Exponentially Harder than Online RLcited by 55
  17. 2020Partial Policy Iteration for L1-Robust Markov Decision Processescited by 50
  18. 2019Learning When-to-Treat Policiescited by 49
  19. 2021Offline Reinforcement Learning for Autonomous Driving with Safety and Exploration Enhancementcited by 29
  20. 2021Towards Theoretical Understandings of Robust Markov Decision Processes: Sample Complexity and Asymptoticscited by 27
  21. 2018State Aggregation Learning from Markov Transition Datacited by 20
  22. 2018Spectral State Compression of Markov Processescited by 18
  23. 2013Scaling Up Robust MDPs by Reinforcement Learningcited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 9; 2024: 9; 2025: 11; 2026: 1.

32 papers.

Sort:

By area:

  1. 2023Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coveragecited by 51
  2. 2023Natural Actor-Critic for Robust Reinforcement Learning with Function Approximationcited by 36
  3. 2022First-order Policy Optimization for Robust Markov Decision Processcited by 32
  4. 2024Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithmscited by 26
  5. 2023Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlationcited by 25
  6. 2024Sample-Efficient Robust Multi-Agent Reinforcement Learning in the Face of Environmental Uncertaintycited by 25
  7. 2023Survival Instinct in Offline Reinforcement Learningcited by 20
  8. 2022Online Policy Optimization for Robust MDPcited by 19
  9. 2024Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learningcited by 18
  10. 2023Soft Robust MDPs and Risk-Sensitive MDPs: Equivalence, Policy Gradient, and Sample Complexitycited by 17
  11. 2023Single-Trajectory Distributionally Robust Reinforcement Learningcited by 16
  12. 2024Upper and Lower Bounds for Distributionally Robust Off-Dynamics Reinforcement Learningcited by 16
20 more
  1. 2023Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounderscited by 13
  2. 2023Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coveragecited by 12
  3. 2024Hybrid Transfer Reinforcement Learning: Provable Sample Efficiency from Shifted-Dynamics Datacited by 8
  4. 2024Robust Offline Reinforcement Learning with Linearly Structured f-Divergence Regularizationcited by 8
  5. 2025ORVIT: Near-Optimal Online Distributionally Robust Reinforcement Learningcited by 6
  6. 2024Roping in Uncertainty: Robustness and Regularization in Markov Gamescited by 5
  7. 2025Linear Mixture Distributionally Robust Markov Decision Processescited by 5
  8. 2025Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Dualitycited by 4
  9. 2023Provably Efficient Offline Reinforcement Learning with Perturbed Data Sourcescited by 3
  10. 2024Policy Gradient for Robust Markov Decision Processescited by 3
  11. 2025Hybrid Cross-domain Robust Reinforcement Learningcited by 3
  12. 2025DR-SAC: Distributionally Robust Soft Actor-Critic for Reinforcement Learning under Uncertaintycited by 3
  13. 2025Towards Optimal Offline Reinforcement Learningcited by 2
  14. 2025Online Robust Reinforcement Learning with General Function Approximationcited by 2
  15. 2025Distributionally Robust Self Paced Curriculum Reinforcement Learningcited by 1
  16. 2025Provably Efficient Sample Complexity for Robust CMDPcited by 1
  17. 2024Efficient Duple Perturbation Robustness in Low-rank MDPscited by 0
  18. 2025DiSA-IQL: Offline Reinforcement Learning for Robust Soft Robot Control under Distribution Shiftscited by 0
  19. 2025Finite-Time Convergence of Single-Trajectory Chi-Square Robust Q-Learning With Linear Function Approximationcited by 0
  20. 2026Interactive Distributionally Robust Multi-Agent Learning with General Function Approximationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Robust Offline Reinforcement Learning for Non-Markovian Decision Processes
  2. 2023Achieving the Asymptotically Optimal Sample Complexity of Offline Reinforcement Learning: A DRO-Based Approach
  3. 2023Offline Reinforcement Learning with Additional Covering Distributions
  4. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian
  5. 2024Sample Complexity of Offline Distributionally Robust Linear Markov Decision Processes
  6. 2022On Instance-Dependent Bounds for Offline Reinforcement Learning with Linear Function Approximation
  7. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation
  8. 2022Offline Reinforcement Learning Under Value and Density-Ratio Realizability: The Power of Gaps