mlsanityData to 14 Oct 2026

Linear Mixture Distributionally Robust Markov Decision Processes

arXiv 2505.18044 ↗First posted 23 May 2025cs.LG cs.AI cs.ROcited by 5

Not on the map. Why?

Abstract (arXiv)

Many real-world decision-making problems face the off-dynamics challenge: the agent learns a policy in a source domain and deploys it in a target domain with different state transitions. The distributionally robust Markov decision process (DRMDP) addresses this challenge by finding a robust policy that performs well under the worst-case environment within a pre-specified uncertainty set of transition dynamics. Its effectiveness heavily hinges on the proper design of these uncertainty sets, based on prior knowledge of the dynamics. In this work, we propose a novel linear mixture DRMDP framework, where the nominal dynamics is assumed to be a linear mixture model. In contrast with existing uncertainty sets directly defined as a ball centered around the nominal kernel, linear mixture DRMDPs define the uncertainty sets based on a ball around the mixture weighting parameter. We show that this new framework provides a more refined representation of uncertainties compared to conventional models based on (s,a)-rectangularity and d-rectangularity, when prior knowledge about the mixture model is present. We propose a meta algorithm for robust policy learning in linear mixture DRMDPs with general f-divergence defined uncertainty sets, and analyze its sample complexities under three divergence metrics instantiations: total variation, Kullback-Leibler, and chi2 divergences. These results establish the statistical learnability of linear mixture DRMDPs, laying the theoretical foundation for future research on this new setting.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 38 of its 63 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

38 of 63 references.

References · 38

  1. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  2. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 413
  3. 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 234
  4. 2019Provably Efficient Exploration in Policy Optimizationcited by 205
  5. 2020Nearly Minimax Optimal Reinforcement Learning for Linear Mixture Markov Decision Processescited by 200
  6. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 164
  7. 2018Assessing Generalization in Deep Reinforcement Learningcited by 133
  8. 2020Provably Efficient Reinforcement Learning for Discounted MDPs with Feature Mappingcited by 102
  9. 2020Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifierscited by 97
  10. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexitycited by 87
  11. 2023The Curious Price of Distributional Robustness in Reinforcement Learning with a Generative Modelcited by 52
  12. 2023Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coveragecited by 51
26 more
  1. 2018Meta Reinforcement Learning with Latent Variable Gaussian Processescited by 44
  2. 2023Improved Sample Complexity Bounds for Distributionally Robust Reinforcement Learningcited by 44
  3. 2025A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Modelscited by 38
  4. 2023Natural Actor-Critic for Robust Reinforcement Learning with Function Approximationcited by 36
  5. 2022Distributionally Robust Offline Reinforcement Learning with Linear Function Approximationcited by 32
  6. 2023Cross-Domain Policy Adaptation via Value-Guided Data Filteringcited by 31
  7. 2021Reward-Free Model-Based Reinforcement Learning with Linear Function Approximationcited by 29
  8. 2024Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithmscited by 26
  9. 2024Distributionally Robust Off-Dynamics Reinforcement Learning: Provable Efficiency with Linear Function Approximationcited by 25
  10. 2024ODRL: A Benchmark for Off-Dynamics Reinforcement Learningcited by 25
  11. 2020Task-Agnostic Online Reinforcement Learning with an Infinite Mixture of Gaussian Processescited by 23
  12. 2024Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learningcited by 18
  13. 2024Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learningcited by 18
  14. 2024Off-Dynamics Reinforcement Learning via Domain Adaptation and Reward Augmented Imitationcited by 17
  15. 2017Mutual Alignment Transfer Learningcited by 16
  16. 2023Policy Gradient Algorithms for Robust MDPs with Non-Rectangular Uncertainty Setscited by 16
  17. 2024Upper and Lower Bounds for Distributionally Robust Off-Dynamics Reinforcement Learningcited by 16
  18. 2024Sample Complexity of Offline Distributionally Robust Linear Markov Decision Processescited by 14
  19. 2024Return Augmented Decision Transformer for Off-Dynamics Reinforcement Learningcited by 13
  20. 2023Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coveragecited by 12
  21. 2024Robust Offline Reinforcement Learning with Linearly Structured f-Divergence Regularizationcited by 8
  22. 2024Improved Algorithm for Adversarial Linear Mixture MDPs with Bandit Feedback and Unknown Transitioncited by 6
  23. 2021Quantifying Multimodality in World Modelscited by 4
  24. 2023Density Steering of Gaussian Mixture Models for Discrete-Time Linear Systemscited by 2
  25. 2024Time-Constrained Robust MDPscited by 2
  26. 2024Stochastic MPC for Finite Gaussian Mixture Disturbances with Guaranteescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025MOBODY: Model Based Off-Dynamics Offline Reinforcement Learningcited by 9
  2. 2025ORVIT: Near-Optimal Online Distributionally Robust Reinforcement Learningcited by 6
  3. 2025Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Dualitycited by 4
  4. 2026Provably Efficient Algorithms for S- and Non-Rectangular Robust MDPs with General Parameterizationcited by 1
  5. 2026Learning Sequential Decisions from Multiple Sources via Group-Robust Markov Decision Processescited by 0