mlsanityData to 13 Oct 2026

Time Discretization-Invariant Safe Action Repetition for Policy Gradient Methods

arXiv 2111.03941 ↗First posted 6 Nov 2021cs.LG cs.AI cs.ROcited by 21

Not on the map. Why?

Abstract (arXiv)

In reinforcement learning, continuous time is often discretized by a time scale δ, to which the resulting performance is known to be highly sensitive. In this work, we seek to find a δ-invariant algorithm for policy gradient (PG) methods, which performs well regardless of the value of δ. We first identify the underlying reasons that cause PG methods to fail as δ → 0, proving that the variance of the PG estimator can diverge to infinity in stochastic environments under a certain assumption of stochasticity. While durative actions or action repetition can be employed to have δ-invariance, previous action repetition methods cannot immediately react to unexpected situations in stochastic environments. We thus propose a novel δ-invariant method named Safe Action Repetition (SAR) applicable to any existing PG algorithm. SAR can handle the stochasticity of environments by adaptively reacting to changes in states during action repetition. We empirically show that our method is not only δ-invariant but also robust to stochasticity, outperforming previous δ-invariant approaches on eight MuJoCo environments with both deterministic and stochastic settings. Our code is available at https://vision.snu.ac.kr/projects/sar.

Key facts. Cited by 21 papers in the map (data through 13 Oct 2026). Builds on 19 of its 36 references that are in the map. First posted on arXiv 6 Nov 2021.

Builds on

19 of 36 references.

References · 19

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,223
  2. 2017Proximal Policy Optimization Algorithmscited by 14,592
  3. 2015Continuous control with deep reinforcement learningcited by 3,290
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,221
  5. 2015Trust Region Policy Optimizationcited by 2,513
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,223
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,940
  8. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,620
  9. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,334
  10. 2020Mastering Atari with Discrete World Modelscited by 996
  11. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 731
  12. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 459
7 more
  1. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  2. 2017Time Limits in Reinforcement Learningcited by 75
  3. 2019Making Deep Q-learning methods robust to time discretizationcited by 51
  4. 2020Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEscited by 32
  5. 2020Improving Robustness via Risk Averse Distributional Reinforcement Learningcited by 28
  6. 2020Control Frequency Adaptation via Action Persistence in Batch Reinforcement Learningcited by 27
  7. 2019Autoregressive Policies for Continuous Control Deep Reinforcement Learningcited by 14

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2022: 5; 2023: 3; 2024: 6; 2025: 6; 2026: 1.

21 papers.

Sort:

By area:

  1. 2023Mastering Diverse Domains through World Modelscited by 1,224
  2. 2022Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problemscited by 20
  3. 2022Dynamic Decision Frequency with Continuous Optionscited by 11
  4. 2024Sublinear Regret for a Class of Continuous-Time Linear-Quadratic Reinforcement Learning Problemscited by 11
  5. 2025Deterministic Policy Gradient for Reinforcement Learning with Continuous Time and Statecited by 7
  6. 2023Subwords as Skills: Tokenization for Sparse-Reward Reinforcement Learningcited by 3
  7. 2024Learning Uncertainty-Aware Temporally-Extended Actionscited by 3
  8. 2022Simultaneously Updating All Persistence Values in Reinforcement Learningcited by 2
  9. 2024Reinforcement Learning with Elastic Time Stepscited by 2
  10. 2024Reinforcement Learning for Jump-Diffusions, with Financial Applicationscited by 2
  11. 2024MOSEAC: Streamlined Variable Time Step Reinforcement Learningcited by 2
  12. 2025Data-Driven Exploration for a Class of Continuous-Time Indefinite Linear--Quadratic Reinforcement Learning Problemscited by 2
9 more
  1. 2025Continuous-Time Value Iteration for Multi-Agent Reinforcement Learningcited by 2
  2. 2025Continuous-Time Reinforcement Learning for Asset-Liability Managementcited by 2
  3. 2022Managing Temporal Resolution in Continuous Value Estimation: A Fundamental Trade-offcited by 1
  4. 2023Actor-Critic with variable time discretization via sustained actionscited by 1
  5. 2024State-Novelty Guided Action Persistence in Deep Reinforcement Learningcited by 1
  6. 2025Policy Optimization for Continuous-time Linear-Quadratic Graphon Mean Field Gamescited by 1
  7. 2022Taming Reachability Analysis of DNN-Controlled Systems via Abstraction-Based Trainingcited by 0
  8. 2025TARC: Time-Adaptive Robotic Controlcited by 0
  9. 2026Reinforcement Learning for Continuous-Time Jump Markov Decision Processes with Applications to Network Dynamic Pricingcited by 0