mlsanityData to 7 Oct 2026

Max-Min Off-Policy Actor-Critic Method Focusing on Worst-Case Robustness to Model Misspecification

arXiv 2211.03413 ↗First posted 7 Nov 2022cs.LG cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

In the field of reinforcement learning, because of the high cost and risk of policy training in the real world, policies are trained in a simulation environment and transferred to the corresponding real-world environment. However, the simulation environment does not perfectly mimic the real-world environment, lead to model misspecification. Multiple studies report significant deterioration of policy performance in a real-world environment. In this study, we focus on scenarios involving a simulation environment with uncertainty parameters and the set of their possible values, called the uncertainty parameter set. The aim is to optimize the worst-case performance on the uncertainty parameter set to guarantee the performance in the corresponding real-world environment. To obtain a policy for the optimization, we propose an off-policy actor-critic approach called the Max-Min Twin Delayed Deep Deterministic Policy Gradient algorithm (M2TD3), which solves a max-min optimization problem using a simultaneous gradient ascent descent approach. Experiments in multi-joint dynamics with contact (MuJoCo) environments show that the proposed method exhibited a worst-case performance superior to several baseline approaches.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 34 of its 64 references that are in the map. First posted on arXiv 7 Nov 2022.

Builds on

34 of 64 references.

References · 34

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2022Generative Adversarial Networkscited by 5,603
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  8. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  9. 2017Robust Adversarial Reinforcement Learningcited by 439
  10. 2018Maximum a Posteriori Policy Optimisationcited by 318
  11. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  12. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 260
22 more
  1. 2018Actor-Attention-Critic for Multi-Agent Reinforcement Learningcited by 259
  2. 2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 235
  3. 2019Active Domain Randomizationcited by 161
  4. 2016EPOpt: Learning Robust Neural Network Policies Using Model Ensemblescited by 136
  5. 2021Robust Reinforcement Learning on State Observations with Learned Optimal Adversarycited by 133
  6. 2017Preparing for the Unknown: Learning a Universal Policy with Online System Identificationcited by 129
  7. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 109
  8. 2017The Numerics of GANscited by 104
  9. 2020Episodic Reinforcement Learning in Finite MDPs: Minimax Lower Bounds Revisitedcited by 104
  10. 2020Batch Value-function Approximation with Only Realizabilitycited by 103
  11. 2020The Complexity of Constrained Min-Max Optimizationcited by 81
  12. 2019Robust Reinforcement Learning for Continuous Control with Model Misspecificationcited by 76
  13. 2018Interaction Matters: A Note on Non-asymptotic Local Convergence of Generative Adversarial Networkscited by 68
  14. 2017Gradient descent GAN optimization is locally stablecited by 65
  15. 2016Connecting Generative Adversarial Networks and Actor-Critic Methodscited by 49
  16. 2020Robust Reinforcement Learning via Adversarial training with Langevin Dynamicscited by 45
  17. 2018Policy Transfer with Strategy Optimizationcited by 41
  18. 2018Augmenting Physical Simulators with Stochastic Neural Networks: Case Study of Planar Pushing and Bouncingcited by 36
  19. 2020Model-based Adversarial Meta-Reinforcement Learningcited by 32
  20. 2018Learning Robust Optionscited by 17
  21. 2020Adversarial Skill Learning for Robust Manipulationcited by 6
  22. 2021Robust Reinforcement Learning under model misspecificationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 5; 2026: 1.

11 papers.

  1. 2023Towards Minimax Optimality of Model-based Robust Reinforcement Learningcited by 21
  2. 2025Robust Gymnasium: A Unified Modular Benchmark for Robust Reinforcement Learningcited by 12
  3. 2025SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfercited by 8
  4. 2025Adversarial Diffusion for Robust Reinforcement Learningcited by 4
  5. 2024Time-Constrained Robust MDPscited by 2
  6. 2024RRLS : Robust Reinforcement Learning Suitecited by 2
  7. 2024Bootstrapping Expectiles in Reinforcement Learningcited by 1
  8. 2024Solving robust MDPs as a sequence of static RL problemscited by 1
  9. 2025Maximum Total Correlation Reinforcement Learningcited by 0
  10. 2025Off-Policy Actor-Critic for Adversarial Observation Robustness: Virtual Alternative Training via Symmetric Policy Evaluationcited by 0
  11. 2026Reward-Preserving Attacks For Robust Reinforcement Learningcited by 0