mlsanityData to 7 Oct 2026

Learning Robust Policy against Disturbance in Transition Dynamics via State-Conservative Policy Optimization

arXiv 2112.10513 ↗First posted 20 Dec 2021cs.LG stat.MLcited by 18

Not on the map. Why?

Abstract (arXiv)

Deep reinforcement learning algorithms can perform poorly in real-world tasks due to the discrepancy between source and target environments. This discrepancy is commonly viewed as the disturbance in transition dynamics. Many existing algorithms learn robust policies by modeling the disturbance and applying it to source environments during training, which usually requires prior knowledge about the disturbance and control of simulators. However, these algorithms can fail in scenarios where the disturbance from target environments is unknown or is intractable to model in simulators. To tackle this problem, we propose a novel model-free actor-critic algorithm -- namely, state-conservative policy optimization (SCPO) -- to learn robust policies without modeling the disturbance in advance. Specifically, SCPO reduces the disturbance in transition dynamics to that in state space and then approximates it by a simple gradient-based regularizer. The appealing features of SCPO include that it is simple to implement and does not require additional knowledge about the disturbance or specially designed simulators. Experiments in several robot control tasks demonstrate that SCPO learns robust policies against the disturbance in transition dynamics.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 14 of its 22 references that are in the map. First posted on arXiv 20 Dec 2021.

Builds on

14 of 22 references.

References · 14

  1. 2015Continuous control with deep reinforcement learningcited by 3,279
  2. 2021Convex optimizationcited by 2,479
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  5. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  6. 2019Solving Rubik's Cube with a Robot Handcited by 699
  7. 2017Robust Adversarial Reinforcement Learningcited by 439
  8. 2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 235
  9. 2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 167
  10. 2016EPOpt: Learning Robust Neural Network Policies Using Model Ensemblescited by 136
  11. 2018Assessing Generalization in Deep Reinforcement Learningcited by 132
  12. 2019Robust Reinforcement Learning for Continuous Control with Model Misspecificationcited by 76
2 more
  1. 2019Wasserstein Robust Reinforcement Learningcited by 50
  2. 2020Robust Reinforcement Learning with Wasserstein Constraintcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 6; 2024: 5; 2025: 5; 2026: 1.

18 papers.

Sort:

By area:

  1. 2023Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coveragecited by 51
  2. 2023Natural Actor-Critic for Robust Reinforcement Learning with Function Approximationcited by 35
  3. 2024Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithmscited by 24
  4. 2024Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learningcited by 17
  5. 2022Provable Sim-to-real Transfer in Continuous Domain with Partial Observationscited by 9
  6. 2023Optimal Transport Perturbations for Safe Reinforcement Learning with Robustness Guaranteescited by 9
  7. 2023Adjustable Robust Reinforcement Learning for Online 3D Bin Packingcited by 9
  8. 2023MICRO: Model-Based Offline Reinforcement Learning with a Conservative Bellman Operatorcited by 9
  9. 2024Robust Deep Reinforcement Learning with Adaptive Adversarial Perturbations in Action Spacecited by 9
  10. 2024Learning to Stop Cut Generation for Efficient Mixed-Integer Linear Programmingcited by 8
  11. 2025Finite-Sample Analysis of Policy Evaluation for Robust Average Reward Reinforcement Learningcited by 6
  12. 2025Dual-Robust Cross-Domain Offline Reinforcement Learning Against Dynamics Shiftscited by 5
6 more
  1. 2025Efficient Q-Learning and Actor-Critic Methods for Robust Average-Reward Reinforcement Learningcited by 2
  2. 2025Generalization in Monitored Markov Decision Processes (Mon-MDPs)cited by 1
  3. 2023Adversarial Policy Optimization in Deep Reinforcement Learningcited by 0
  4. 2024Lipschitz-Regularized Critics Lead to Policy Robustness Against Transition Dynamics Uncertaintycited by 0
  5. 2025Robust Reinforcement Learning in Finance: Modeling Market Impact with Elliptic Uncertainty Setscited by 0
  6. 2026Beyond Coefficients: Forecast-Necessity Testing for Interpretable Causal Discovery in Nonlinear Time-Series Modelscited by 0