# Time Discretization-Invariant Safe Action Repetition for Policy Gradient Methods

arXiv: https://arxiv.org/abs/2111.03941 · HTML: https://mlsanity.com/paper/2111.03941/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 36 references that are in the map.
- First posted on arXiv 6 Nov 2021.

## Builds on (most cited first, 19 of 19)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [Deep Recurrent Q-Learning for Partially Observable MDPs](https://mlsanity.com/paper/1507.06527/) (2015)
- [Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates](https://mlsanity.com/paper/1610.00633/) (2016)
- [Time Limits in Reinforcement Learning](https://mlsanity.com/paper/1712.00378/) (2017)
- [Making Deep Q-learning methods robust to time discretization](https://mlsanity.com/paper/1901.09732/) (2019)
- [Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEs](https://mlsanity.com/paper/2006.16210/) (2020)
- [Control Frequency Adaptation via Action Persistence in Batch Reinforcement Learning](https://mlsanity.com/paper/2002.06836/) (2020)
- [Improving Robustness via Risk Averse Distributional Reinforcement Learning](https://mlsanity.com/paper/2005.00585/) (2020)
- [Autoregressive Policies for Continuous Control Deep Reinforcement Learning](https://mlsanity.com/paper/1903.11524/) (2019)

## Built on by (most cited first, 23 of 23)

- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problems](https://mlsanity.com/paper/2211.00617/) (2022)
- [Dynamic Decision Frequency with Continuous Options](https://mlsanity.com/paper/2212.04407/) (2022)
- [Sublinear Regret for a Class of Continuous-Time Linear-Quadratic Reinforcement Learning Problems](https://mlsanity.com/paper/2407.17226/) (2024)
- [Deterministic Policy Gradient for Reinforcement Learning with Continuous Time and State](https://mlsanity.com/paper/2509.23711/) (2025)
- [Subwords as Skills: Tokenization for Sparse-Reward Reinforcement Learning](https://mlsanity.com/paper/2309.04459/) (2023)
- [Learning Uncertainty-Aware Temporally-Extended Actions](https://mlsanity.com/paper/2402.05439/) (2024)
- [Reinforcement Learning with Elastic Time Steps](https://mlsanity.com/paper/2402.14961/) (2024)
- [Data-Driven Exploration for a Class of Continuous-Time Indefinite Linear--Quadratic Reinforcement Learning Problems](https://mlsanity.com/paper/2507.00358/) (2025)
- [Simultaneously Updating All Persistence Values in Reinforcement Learning](https://mlsanity.com/paper/2211.11620/) (2022)
- [Reinforcement Learning for Jump-Diffusions, with Financial Applications](https://mlsanity.com/paper/2405.16449/) (2024)
- [MOSEAC: Streamlined Variable Time Step Reinforcement Learning](https://mlsanity.com/paper/2406.01521/) (2024)
- [Continuous-Time Value Iteration for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2509.09135/) (2025)
- [Continuous-Time Reinforcement Learning for Asset-Liability Management](https://mlsanity.com/paper/2509.23280/) (2025)
- [Managing Temporal Resolution in Continuous Value Estimation: A Fundamental Trade-off](https://mlsanity.com/paper/2212.08949/) (2022)
- [Actor-Critic with variable time discretization via sustained actions](https://mlsanity.com/paper/2308.04299/) (2023)
- [State-Novelty Guided Action Persistence in Deep Reinforcement Learning](https://mlsanity.com/paper/2409.05433/) (2024)
- [Policy Optimization for Continuous-time Linear-Quadratic Graphon Mean Field Games](https://mlsanity.com/paper/2506.05894/) (2025)
- [TARC: Time-Adaptive Robotic Control](https://mlsanity.com/paper/2510.23176/) (2025)
- [Taming Reachability Analysis of DNN-Controlled Systems via Abstraction-Based Training](https://mlsanity.com/paper/2211.11127/) (2022)
- [Variable Time Step Reinforcement Learning for Robotic Applications](https://mlsanity.com/paper/2407.00290/) (2024)
- [A hierarchical resource-efficient deep policy gradient method for continuous-time optimal control problems](https://mlsanity.com/paper/2502.14141/) (2025)
- [Reinforcement Learning for Continuous-Time Jump Markov Decision Processes with Applications to Network Dynamic Pricing](https://mlsanity.com/paper/2608.20680/) (2026)

## Abstract (arXiv)

In reinforcement learning, continuous time is often discretized by a time scale δ, to which the resulting performance is known to be highly sensitive. In this work, we seek to find a δ-invariant algorithm for policy gradient (PG) methods, which performs well regardless of the value of δ. We first identify the underlying reasons that cause PG methods to fail as δ → 0, proving that the variance of the PG estimator can diverge to infinity in stochastic environments under a certain assumption of stochasticity. While durative actions or action repetition can be employed to have δ-invariance, previous action repetition methods cannot immediately react to unexpected situations in stochastic environments. We thus propose a novel δ-invariant method named Safe Action Repetition (SAR) applicable to any existing PG algorithm. SAR can handle the stochasticity of environments by adaptively reacting to changes in states during action repetition. We empirically show that our method is not only δ-invariant but also robust to stochasticity, outperforming previous δ-invariant approaches on eight MuJoCo environments with both deterministic and stochastic settings. Our code is available at https://vision.snu.ac.kr/projects/sar.