# Actor-Critic with variable time discretization via sustained actions

arXiv: https://arxiv.org/abs/2308.04299 · HTML: https://mlsanity.com/paper/2308.04299/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 35 references that are in the map.
- First posted on arXiv 8 Aug 2023.

## Builds on (most cited first, 18 of 18)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Reinforcement Learning in Healthcare: A Survey](https://mlsanity.com/paper/1908.08796/) (2019)
- [The Limits and Potentials of Deep Learning for Robotics](https://mlsanity.com/paper/1804.06557/) (2018)
- [Making Deep Q-learning methods robust to time discretization](https://mlsanity.com/paper/1901.09732/) (2019)
- [Learning to Repeat: Fine Grained Action Repetition for Deep Reinforcement Learning](https://mlsanity.com/paper/1702.06054/) (2017)
- [Deep Reinforcement Learning for the Control of Robotic Manipulation: A Focussed Mini-Review](https://mlsanity.com/paper/2102.04148/) (2021)
- [Control Frequency Adaptation via Action Persistence in Batch Reinforcement Learning](https://mlsanity.com/paper/2002.06836/) (2020)
- [TempoRL: Learning When to Act](https://mlsanity.com/paper/2106.05262/) (2021)
- [Time Discretization-Invariant Safe Action Repetition for Policy Gradient Methods](https://mlsanity.com/paper/2111.03941/) (2021)
- [Hierarchical Reinforcement Learning with Timed Subgoals](https://mlsanity.com/paper/2112.03100/) (2021)
- [An Analysis of Frame-skipping in Reinforcement Learning](https://mlsanity.com/paper/2102.03718/) (2021)
- [ACERAC: Efficient reinforcement learning in fine time discretization](https://mlsanity.com/paper/2104.04004/) (2021)
- [A framework for reinforcement learning with autocorrelated actions](https://mlsanity.com/paper/2009.04777/) (2020)

## Built on by (most cited first, 1 of 1)

- [State-Novelty Guided Action Persistence in Deep Reinforcement Learning](https://mlsanity.com/paper/2409.05433/) (2024)

## Abstract (arXiv)

Reinforcement learning (RL) methods work in discrete time. In order to apply RL to inherently continuous problems like robotic control, a specific time discretization needs to be defined. This is a choice between sparse time control, which may be easier to train, and finer time control, which may allow for better ultimate performance. In this work, we propose SusACER, an off-policy RL algorithm that combines the advantages of different time discretization settings. Initially, it operates with sparse time discretization and gradually switches to a fine one. We analyze the effects of the changing time discretization in robotic control environments: Ant, HalfCheetah, Hopper, and Walker2D. In all cases our proposed algorithm outperforms state of the art.