mlsanityData to 13 Oct 2026

Single-Trajectory Distributionally Robust Reinforcement Learning

arXiv 2301.11721 ↗First posted 27 Jan 2023stat.ML cs.AI cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

To mitigate the limitation that the classical reinforcement learning (RL) framework heavily relies on identical training and test environments, Distributionally Robust RL (DRRL) has been proposed to enhance performance across a range of environments, possibly including unknown test environments. As a price for robustness gain, DRRL involves optimizing over a set of distributions, which is inherently more challenging than optimizing over a fixed distribution in the non-robust case. Existing DRRL algorithms are either model-based or fail to learn from a single sample trajectory. In this paper, we design a first fully model-free DRRL algorithm, called distributionally robust Q-learning with single trajectory (DRQ). We delicately design a multi-timescale framework to fully utilize each incrementally arriving sample and directly learn the optimal distributionally robust policy without modelling the environment, thus the algorithm can be trained along a single trajectory in a model-free fashion. Despite the algorithm's complexity, we provide asymptotic convergence guarantees by generalizing classical stochastic approximation tools. Comprehensive experimental results demonstrate the superior robustness and sample complexity of our proposed algorithm, compared to non-robust methods and other robust RL algorithms.

Key facts. Cited by 15 papers in the map (data through 13 Oct 2026). Builds on 23 of its 65 references that are in the map. First posted on arXiv 27 Jan 2023.

Builds on

23 of 65 references.

References · 23

  1. 2017Proximal Policy Optimization Algorithmscited by 14,592
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,336
  3. 2015Continuous control with deep reinforcement learningcited by 3,290
  4. 2015Domain-Adversarial Training of Neural Networkscited by 3,030
  5. 2016OpenAI Gymcited by 1,694
  6. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,511
  7. 2017Ensemble Adversarial Training: Attacks and Defensescited by 839
  8. 2018Learning Models with Uniform Performance via Distributionally Robust Optimizationcited by 335
  9. 2018Maximum a Posteriori Policy Optimisationcited by 320
  10. 2021Online Robust Reinforcement Learning with Model Uncertaintycited by 95
  11. 2022Robust Reinforcement Learning using Offline Datacited by 95
  12. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexitycited by 85
11 more
  1. 2021Sample Complexity of Robust Reinforcement Learning with a Generative Modelcited by 82
  2. 2020Learning and Planning in Average-Reward Markov Decision Processescited by 54
  3. 2017Reinforcement Learning under Model Mismatchcited by 52
  4. 2019Wasserstein Robust Reinforcement Learningcited by 50
  5. 2020Partial Policy Iteration for L1-Robust Markov Decision Processescited by 49
  6. 2018Soft-Robust Actor-Critic Policy-Gradientcited by 37
  7. 2022Distributionally Robust Offline Reinforcement Learning with Linear Function Approximationcited by 31
  8. 2018Wasserstein Distributionally Robust Stochastic Control: A Data-Driven Approachcited by 27
  9. 2021Towards Theoretical Understandings of Robust Markov Decision Processes: Sample Complexity and Asymptoticscited by 27
  10. 2023Robust Average-Reward Markov Decision Processescited by 22
  11. 2021Model-Free Risk-Sensitive Reinforcement Learningcited by 7

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 4; 2024: 5; 2025: 5; 2026: 1.

15 papers.

Sort:

By area:

  1. 2023The Curious Price of Distributional Robustness in Reinforcement Learning with a Generative Modelcited by 50
  2. 2023Natural Actor-Critic for Robust Reinforcement Learning with Function Approximationcited by 36
  3. 2023Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlationcited by 25
  4. 2024Sample-Efficient Robust Multi-Agent Reinforcement Learning in the Face of Environmental Uncertaintycited by 24
  5. 2023Distributionally Robust Model-based Reinforcement Learning with Large State Spacescited by 17
  6. 2024Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learningcited by 17
  7. 2024Sample Complexity of Offline Distributionally Robust Linear Markov Decision Processescited by 12
  8. 2024Model-Free Robust Reinforcement Learning with Sample Complexity Analysiscited by 9
  9. 2025Efficient Policy Optimization in Robust Constrained MDPs with Iteration Complexity Guaranteescited by 6
  10. 2025Model-Free Robust Average-Reward Reinforcement Learning with Sample Complexity Analysiscited by 3
  11. 2024Model-Free Robust φ-Divergence Reinforcement Learning Using Both Offline and Online Datacited by 2
  12. 2025Provably Efficient Sample Complexity for Robust CMDPcited by 1
3 more
  1. 2025Online Robust Reinforcement Learning with General Function Approximationcited by 1
  2. 2025Finite-Time Convergence of Single-Trajectory Chi-Square Robust Q-Learning With Linear Function Approximationcited by 0
  3. 2026Central Limit Theorem for Two-Time-Scale Approximate Distributionally Robust RLcited by 0