mlsanityData to 7 Oct 2026

Distributionally Robust Off-Dynamics Reinforcement Learning: Provable Efficiency with Linear Function Approximation

arXiv 2402.15399 ↗First posted 23 Feb 2024cs.LGcited by 25

Not on the map. Why?

Abstract (arXiv)

We study off-dynamics Reinforcement Learning (RL), where the policy is trained on a source domain and deployed to a distinct target domain. We aim to solve this problem via online distributionally robust Markov decision processes (DRMDPs), where the learning algorithm actively interacts with the source domain while seeking the optimal performance under the worst possible dynamics that is within an uncertainty set of the source domain's transition kernel. We provide the first study on online DRMDPs with function approximation for off-dynamics RL. We find that DRMDPs' dual formulation can induce nonlinearity, even when the nominal transition kernel is linear, leading to error propagation. By designing a d-rectangular uncertainty set using the total variation distance, we remove this additional nonlinearity and bypass the error propagation. We then introduce DR-LSVI-UCB, the first provably efficient online DRMDP algorithm for off-dynamics RL with function approximation, and establish a polynomial suboptimality bound that is independent of the state and action space sizes. Our work makes the first step towards a deeper understanding of the provable efficiency of online DRMDPs with linear function approximation. Finally, we substantiate the performance and robustness of DR-LSVI-UCB through different numerical experiments.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 37 of its 71 references that are in the map. First posted on arXiv 23 Feb 2024.

Builds on

37 of 71 references.

References · 37

  1. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  2. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  3. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  4. 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
  5. 2017Robust Adversarial Reinforcement Learningcited by 439
  6. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 388
  7. 2018A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximationcited by 225
  8. 2019Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Boundcited by 199
  9. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 135
  10. 2021Robust Reinforcement Learning on State Observations with Learned Optimal Adversarycited by 133
  11. 2018Assessing Generalization in Deep Reinforcement Learningcited by 132
  12. 2018Generalization and Regularization in DQNcited by 117
25 more
  1. 2019Sample Complexity of Reinforcement Learning using Linearly Combined Model Ensemblescited by 100
  2. 2019Frequentist Regret Bounds for Randomized Least-Squares Value Iterationcited by 100
  3. 2022Robust Reinforcement Learning using Offline Datacited by 95
  4. 2020Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifierscited by 93
  5. 2020On Reward-Free Reinforcement Learning with Linear Function Approximationcited by 89
  6. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexitycited by 85
  7. 2021Sample Complexity of Robust Reinforcement Learning with a Generative Modelcited by 82
  8. 2020Logarithmic Regret for Reinforcement Learning with Linear Function Approximationcited by 79
  9. 2019Robust Reinforcement Learning for Continuous Control with Model Misspecificationcited by 76
  10. 2022Nearly Minimax Optimal Reinforcement Learning for Linear Markov Decision Processescited by 57
  11. 2023Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coveragecited by 51
  12. 2023The Curious Price of Distributional Robustness in Reinforcement Learning with a Generative Modelcited by 50
  13. 2022Reward-Free RL is No Harder Than Reward-Aware RL in Linear Markov Decision Processescited by 49
  14. 2020Learning Infinite-horizon Average-reward MDPs with Linear Function Approximationcited by 48
  15. 2023Improved Sample Complexity Bounds for Distributionally Robust Reinforcement Learningcited by 44
  16. 2018Soft-Robust Actor-Critic Policy-Gradientcited by 37
  17. 2015Distributionally Robust Counterpart in Markov Decision Processescited by 32
  18. 2019A Bayesian Approach to Robust Reinforcement Learningcited by 32
  19. 2023Provable and Practical: Efficient Exploration in Reinforcement Learning via Langevin Monte Carlocited by 29
  20. 2021SimGAN: Hybrid Simulator Identification for Domain Adaptation via Adversarial Reinforcement Learningcited by 27
  21. 2022Online Policy Optimization for Robust MDPcited by 19
  22. 2017Mutual Alignment Transfer Learningcited by 16
  23. 2023Wasserstein Distributionally Robust Policy Evaluation and Learning for Contextual Banditscited by 9
  24. 2023On Estimation of Optimal Dynamic Treatment Regimes with Multiple Treatments for Survival Data-With Application to Colorectal Cancer Studycited by 2
  25. 2018Constructing Stabilized Dynamic Treatment Regimes for Censored Datacited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 11; 2026: 7.

25 papers.

Sort:

By area:

  1. 2024Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithmscited by 24
  2. 2024Sample-Efficient Robust Multi-Agent Reinforcement Learning in the Face of Environmental Uncertaintycited by 24
  3. 2024Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learningcited by 17
  4. 2024Upper and Lower Bounds for Distributionally Robust Off-Dynamics Reinforcement Learningcited by 15
  5. 2024Off-Dynamics Reinforcement Learning via Domain Adaptation and Reward Augmented Imitationcited by 15
  6. 2024Return Augmented Decision Transformer for Off-Dynamics Reinforcement Learningcited by 13
  7. 2024Robust Offline Reinforcement Learning with Linearly Structured f-Divergence Regularizationcited by 8
  8. 2025MOBODY: Model Based Off-Dynamics Offline Reinforcement Learningcited by 8
  9. 2025Linear Mixture Distributionally Robust Markov Decision Processescited by 5
  10. 2025ORVIT: Near-Optimal Online Distributionally Robust Reinforcement Learningcited by 5
  11. 2025Dual-Robust Cross-Domain Offline Reinforcement Learning Against Dynamics Shiftscited by 5
  12. 2025Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Dualitycited by 4
13 more
  1. 2025Pessimism Principle Can Be Effective: Towards a Framework for Zero-Shot Transfer Reinforcement Learningcited by 2
  2. 2025Bi-Level Reinforcement Learning Pathway for Sim-to-Real Optimalitycited by 1
  3. 2025Distributionally Robust Online Markov Game with Linear Function Approximationcited by 1
  4. 2025Online Robust Reinforcement Learning with General Function Approximationcited by 1
  5. 2026Localized Dynamics-Aware Domain Adaption for Off-Dynamics Offline Reinforcement Learningcited by 1
  6. 2026Online Robust Reinforcement Learning Through Monte-Carlo Planningcited by 1
  7. 2025Finite-Time Convergence of Single-Trajectory Chi-Square Robust Q-Learning With Linear Function Approximationcited by 0
  8. 2025Robust Reinforcement Learning in Finance: Modeling Market Impact with Elliptic Uncertainty Setscited by 0
  9. 2026Distributionally Robust Cooperative Multi-Agent Reinforcement Learning via Robust Value Factorizationcited by 0
  10. 2026Cross-Domain Energy-Guided Diffusion Generation for Off-Dynamics Reinforcement Learningcited by 0
  11. 2026Transferable Reinforcement Learning via Probabilistic Latent Embeddings and Dynamic Policy Adaptation for Sim-to-Real Deploymentcited by 0
  12. 2026Diagnose, Recover, Certify: Task Readiness under Hidden Dynamics Changescited by 0
  13. 2026Linear Programming Representations and Strongly Polynomial Algorithms for Robust Markov Decision Processescited by 0