mlsanityData to 14 Oct 2026

Natural Actor-Critic for Robust Reinforcement Learning with Function Approximation

arXiv 2307.08875 ↗First posted 17 Jul 2023cs.LG cs.RO math.OCcited by 36

Not on the map. Why?

Cited by 27 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We study robust reinforcement learning (RL) with the goal of determining a well-performing policy that is robust against model mismatch between the training simulator and the testing environment. Previous policy-based robust RL algorithms mainly focus on the tabular setting under uncertainty sets that facilitate robust policy evaluation, but are no longer tractable when the number of states scales up. To this end, we propose two novel uncertainty set formulations, one based on double sampling and the other on an integral probability metric. Both make large-scale robust RL tractable even when one only has access to a simulator. We propose a robust natural actor-critic (RNAC) approach that incorporates the new uncertainty sets and employs function approximation. We provide finite-time convergence guarantees for the proposed RNAC algorithm to the optimal robust policy within the function approximation error. Finally, we demonstrate the robust performance of the policy learned by our proposed RNAC approach in multiple MuJoCo environments and a real-world TurtleBot navigation task.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Builds on 41 of its 70 references that are in the map. First posted on arXiv 17 Jul 2023.

Builds on

41 of 70 references.

References · 41

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2015Continuous control with deep reinforcement learningcited by 3,407
  5. 2015Trust Region Policy Optimizationcited by 2,580
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  7. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  8. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
  9. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  10. 2018Learning Models with Uniform Performance via Distributionally Robust Optimizationcited by 335
  11. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
  12. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 176
29 more
  1. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  2. 2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 174
  3. 2018The Limits and Potentials of Deep Learning for Roboticscited by 104
  4. 2021Online Robust Reinforcement Learning with Model Uncertaintycited by 96
  5. 2022Robust Reinforcement Learning using Offline Datacited by 96
  6. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexitycited by 87
  7. 2021Sample Complexity of Robust Reinforcement Learning with a Generative Modelcited by 83
  8. 2022Policy Gradient Method For Robust Reinforcement Learningcited by 69
  9. 2019Finite-Sample Analysis of Nonlinear Stochastic Approximation with Applications in Reinforcement Learningcited by 67
  10. 2019Wasserstein Robust Reinforcement Learningcited by 54
  11. 2017Reinforcement Learning under Model Mismatchcited by 52
  12. 2023Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coveragecited by 51
  13. 2022Dichotomy of Control: Separating What You Can Control from What You Cannotcited by 47
  14. 2023Improved Sample Complexity Bounds for Distributionally Robust Reinforcement Learningcited by 44
  15. 2019Beyond Confidence Regions: Tight Bayesian Ambiguity Sets for Robust MDPscited by 40
  16. 2018Soft-Robust Actor-Critic Policy-Gradientcited by 37
  17. 2023A Finite Sample Complexity Bound for Distributionally Robust Q-learningcited by 36
  18. 2022Distributionally Robust Offline Reinforcement Learning with Linear Function Approximationcited by 32
  19. 2022First-order Policy Optimization for Robust Markov Decision Processcited by 32
  20. 2023Policy Gradient for Rectangular Robust Markov Decision Processescited by 31
  21. 2020Scalable First-Order Methods for Robust MDPscited by 25
  22. 2021Policy Optimization for Constrained MDPs with Provable Fast Global Convergencecited by 23
  23. 2021Finite-Sample Analysis of Off-Policy Natural Actor-Critic with Linear Function Approximationcited by 22
  24. 2022On the Global Convergence Rates of Decentralized Softmax Gradient Play in Markov Potential Gamescited by 22
  25. 2022Efficient Policy Iteration for Robust Markov Decision Processes via Regularizationcited by 20
  26. 2021Learning Robust Policy against Disturbance in Transition Dynamics via State-Conservative Policy Optimizationcited by 18
  27. 2022Linear Convergence for Natural Policy Gradient with Log-linear Policy Parametrizationcited by 17
  28. 2023Single-Trajectory Distributionally Robust Reinforcement Learningcited by 16
  29. 2022Anchor-Changing Regularized Natural Policy Gradient for Multi-Objective Reinforcement Learningcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 7; 2025: 19; 2026: 8.

36 papers.

Sort:

By area:

  1. 2024Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithmscited by 26
  2. 2024Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learningcited by 18
  3. 2025Robust LLM Alignment via Distributionally Robust Direct Preference Optimizationcited by 18
  4. 2023Provably Fast Convergence of Independent Natural Policy Gradient for Markov Potential Gamescited by 13
  5. 2025Offline Reinforcement Learning in Large State Spaces: Algorithms and Guaranteescited by 10
  6. 2025Finite-Sample Analysis of Policy Evaluation for Robust Average Reward Reinforcement Learningcited by 7
  7. 2025Efficient Policy Optimization in Robust Constrained MDPs with Iteration Complexity Guaranteescited by 6
  8. 2024Provable Policy Gradient Methods for Average-Reward Markov Potential Gamescited by 5
  9. 2025Linear Mixture Distributionally Robust Markov Decision Processescited by 5
  10. 2024Model-Free Robust φ-Divergence Reinforcement Learning Using Both Offline and Online Datacited by 4
  11. 2024Towards Unraveling and Improving Generalization in World Modelscited by 4
  12. 2025Efficient Q-Learning and Actor-Critic Methods for Robust Average-Reward Reinforcement Learningcited by 4
24 more
  1. 2025Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Dualitycited by 4
  2. 2024Policy Gradient for Robust Markov Decision Processescited by 3
  3. 2025Dual Formulation for Non-Rectangular Lp Robust Markov Decision Processescited by 3
  4. 2025DR-SAC: Distributionally Robust Soft Actor-Critic for Reinforcement Learning under Uncertaintycited by 3
  5. 2025Session-Level Dynamic Ad Load Optimization using Offline Robust Reinforcement Learningcited by 2
  6. 2025Pessimism Principle Can Be Effective: Towards a Framework for Zero-Shot Transfer Reinforcement Learningcited by 2
  7. 2025Online Robust Reinforcement Learning with General Function Approximationcited by 2
  8. 2026RE-SAC: Disentangling aleatoric and epistemic risks in bus fleet control: A stable and robust ensemble DRL approachcited by 2
  9. 2026Robust Adversarial Policy Optimization Under Dynamics Uncertaintycited by 2
  10. 2024Linear Convergence of Independent Natural Policy Gradient in Games with Entropy Regularizationcited by 1
  11. 2025Online Bayesian Risk-Averse Reinforcement Learningcited by 1
  12. 2025Diffusion Policies with Offline and Inverse Reinforcement Learning for Promoting Physical Activity in Older Adults Using Wearable Sensorscited by 1
  13. 2025Distributionally Robust Self Paced Curriculum Reinforcement Learningcited by 1
  14. 2025Primal-Only Actor Critic Algorithm for Robust Constrained Average Cost MDPscited by 1
  15. 2026BAPR: Bayesian amnesic piecewise-robust reinforcement learning for non-stationary continuous controlcited by 1
  16. 2023Game-Theoretic Robust Reinforcement Learning Handles Temporally-Coupled Perturbationscited by 0
  17. 2025Trajectory Entropy Reinforcement Learning for Robust Robot Motor Skill Learningcited by 0
  18. 2025Finite-Time Convergence of Single-Trajectory Chi-Square Robust Q-Learning With Linear Function Approximationcited by 0
  19. 2025Robust Reinforcement Learning in Finance: Modeling Market Impact with Elliptic Uncertainty Setscited by 0
  20. 2026When Backdoors Meet Partial Observability: Attacking Real-World Reinforcement Learningcited by 0
  21. 2026Post-Hoc Robustness for Model-Based Reinforcement Learningcited by 0
  22. 2026Entropy Regularization Improves Policy Robustness in Continuous-Time Reinforcement Learningcited by 0
  23. 2026Robust Peak-cost Constrained Reinforcement Learningcited by 0
  24. 2026Arithmetic Actor Heads and Training Stabilization for Out-of-Distribution Reinforcement Learningcited by 0