mlsanityData to 14 Oct 2026

Finite-Time Convergence of Single-Trajectory Chi-Square Robust Q-Learning With Linear Function Approximation

arXiv 2510.01721 ↗First posted 2 Oct 2025cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Distributionally robust reinforcement learning seeks policies that remain effective when the deployment environment differs from the one that generated the training data. We study model-free robust Q-learning with chi2 uncertainty sets and linear function approximation, using data from a single trajectory of an unknown nominal MDP. Evaluating the chi2 robust Bellman target introduces the square root of a conditional second moment, which cannot be estimated unbiasedly from one transition, while the projected robust Bellman operator need not be contractive. We address these obstacles through a variational reformulation of the robust Bellman target and a blockwise frozen-target scheme, and establish a finite-time error bound relative to the optimal robust Q-function for every γ∈(0,1). A neural-network experiment illustrates how the variational target can be used in a continuous-state nonlinear-control task.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 26 of its 36 references that are in the map. First posted on arXiv 2 Oct 2025.

Builds on

26 of 36 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025ORVIT: Near-Optimal Online Distributionally Robust Reinforcement Learning5 Aug
  2. 2025Model-Free Robust Average-Reward Reinforcement Learning with Sample Complexity Analysis18 May
  3. 2025Sample Complexity of Distributionally Robust Average-Reward Reinforcement Learning15 May
  4. 2025Finite-Sample Analysis of Policy Evaluation for Robust Average Reward Reinforcement Learning24 Feb
  5. 2024Model-Free Robust Reinforcement Learning with Sample Complexity Analysis24 Jun
  6. 2024Model-Free Robust φ-Divergence Reinforcement Learning Using Both Offline and Online Data8 May
  7. 2024Distributionally Robust Off-Dynamics Reinforcement Learning: Provable Efficiency with Linear Function Approximation23 Feb

Other references · 19

  1. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  2. 2016OpenAI Gymcited by 1,736
  3. 2018Learning Models with Uniform Performance via Distributionally Robust Optimizationcited by 335
  4. 2019Finite-Time Error Bounds For Linear Stochastic Approximation and TD Learningcited by 166
  5. 2016Variance-based regularization with convex objectivescited by 155
  6. 2021Online Robust Reinforcement Learning with Model Uncertaintycited by 96
  7. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexitycited by 87
  8. 2021Sample Complexity of Robust Reinforcement Learning with a Generative Modelcited by 83
  9. 2017Reinforcement Learning under Model Mismatchcited by 52
  10. 2023The Curious Price of Distributional Robustness in Reinforcement Learning with a Generative Modelcited by 52
  11. 2023Improved Sample Complexity Bounds for Distributionally Robust Reinforcement Learningcited by 44
  12. 2023A Finite Sample Complexity Bound for Distributionally Robust Q-learningcited by 36
7 more
  1. 2023Natural Actor-Critic for Robust Reinforcement Learning with Function Approximationcited by 36
  2. 2022Distributionally Robust Offline Reinforcement Learning with Linear Function Approximationcited by 32
  3. 2022First-order Policy Optimization for Robust Markov Decision Processcited by 32
  4. 2022Target Network and Truncation Overcome The Deadly Triad in Q-Learningcited by 31
  5. 2023Sample Complexity of Variance-reduced Distributionally Robust Q-learningcited by 29
  6. 2023Single-Trajectory Distributionally Robust Reinforcement Learningcited by 16
  7. 2024Robust Offline Reinforcement Learning with Linearly Structured f-Divergence Regularizationcited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficient Q-Learning and Actor-Critic Methods for Robust Average-Reward Reinforcement Learning
  2. 2025Distributionally Robust Deep Q-Learning
  3. 2024Towards Optimal Adversarial Robust Q-learning with Bellman Infinity-error
  4. 2024Q-Learning under Finite Model Uncertainty
  5. 2024Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learning
  6. 2026Central Limit Theorem for Two-Time-Scale Approximate Distributionally Robust RL
  7. 2024Time-Constrained Robust MDPs
  8. 2024Zero-Sum Positional Differential Games as a Framework for Robust Reinforcement Learning: Deep Q-Learning Approach