# Dynamic Decision Frequency with Continuous Options

arXiv: https://arxiv.org/abs/2212.04407 · HTML: https://mlsanity.com/paper/2212.04407/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 26 references that are in the map.
- First posted on arXiv 6 Dec 2022.

## Builds on (most cited first, 16 of 16)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [When Waiting is not an Option : Learning Options with a Deliberation Cost](https://mlsanity.com/paper/1709.04571/) (2017)
- [Making Deep Q-learning methods robust to time discretization](https://mlsanity.com/paper/1901.09732/) (2019)
- [DAC: The Double Actor-Critic Architecture for Learning Options](https://mlsanity.com/paper/1904.12691/) (2019)
- [Unifying task specification in reinforcement learning](https://mlsanity.com/paper/1609.01995/) (2016)
- [Continuous-Time Model-Based Reinforcement Learning](https://mlsanity.com/paper/2102.04764/) (2021)
- [Learning to Repeat: Fine Grained Action Repetition for Deep Reinforcement Learning](https://mlsanity.com/paper/1702.06054/) (2017)
- [Setting up a Reinforcement Learning Task with a Real-World Robot](https://mlsanity.com/paper/1803.07067/) (2018)
- [Thinking While Moving: Deep Reinforcement Learning with Concurrent Control](https://mlsanity.com/paper/2004.06089/) (2020)
- [Control Frequency Adaptation via Action Persistence in Batch Reinforcement Learning](https://mlsanity.com/paper/2002.06836/) (2020)
- [Time Discretization-Invariant Safe Action Repetition for Policy Gradient Methods](https://mlsanity.com/paper/2111.03941/) (2021)
- [Asynchronous Reinforcement Learning for Real-Time Control of Physical Robots](https://mlsanity.com/paper/2203.12759/) (2022)
- [Locally Persistent Exploration in Continuous Control Tasks with Sparse Rewards](https://mlsanity.com/paper/2012.13658/) (2020)

## Built on by (most cited first, 13 of 13)

- [Revisiting Sparse Rewards for Goal-Reaching Reinforcement Learning](https://mlsanity.com/paper/2407.00324/) (2024)
- [Enabling Realtime Reinforcement Learning at Scale with Staggered Asynchronous Inference](https://mlsanity.com/paper/2412.14355/) (2024)
- [Reinforcement Learning with Elastic Time Steps](https://mlsanity.com/paper/2402.14961/) (2024)
- [MOSEAC: Streamlined Variable Time Step Reinforcement Learning](https://mlsanity.com/paper/2406.01521/) (2024)
- [An Idiosyncrasy of Time-discretization in Reinforcement Learning](https://mlsanity.com/paper/2406.14951/) (2024)
- [TARC: Time-Adaptive Robotic Control](https://mlsanity.com/paper/2510.23176/) (2025)
- [When to Plan: Learning to Select Between Reactive Control and Deliberative Planning](https://mlsanity.com/paper/2607.16421/) (2026)
- [Variable Time Step Reinforcement Learning for Robotic Applications](https://mlsanity.com/paper/2407.00290/) (2024)
- [A Robust Model-Based Approach for Continuous-Time Policy Evaluation with Unknown L'evy Process Dynamics](https://mlsanity.com/paper/2504.01482/) (2025)
- [Learning Multi-Timescale Interventions under Safety and Resource Constraints](https://mlsanity.com/paper/2508.03875/) (2025)
- [Timescale Separation Enables Deep Reinforcement Learning Control of Rotating Detonation Engine Mode Transitions](https://mlsanity.com/paper/2604.14398/) (2026)
- [Action Chunking Proximal Policy Optimization with Feedback Correction](https://mlsanity.com/paper/2609.36250/) (2026)
- [Chronos Policy: Learning Robot Policies on Physical Time](https://mlsanity.com/paper/2610.13929/) (2026)

## Abstract (arXiv)

In classic reinforcement learning algorithms, agents make decisions at discrete and fixed time intervals. The duration between decisions becomes a crucial hyperparameter, as setting it too short may increase the problem's difficulty by requiring the agent to make numerous decisions to achieve its goal while setting it too long can result in the agent losing control over the system. However, physical systems do not necessarily require a constant control frequency, and for learning agents, it is often preferable to operate with a low frequency when possible and a high frequency when necessary. We propose a framework called Continuous-Time Continuous-Options (CTCO), where the agent chooses options as sub-policies of variable durations. These options are time-continuous and can interact with the system at any desired frequency providing a smooth change of actions. We demonstrate the effectiveness of CTCO by comparing its performance to classical RL and temporal-abstraction RL methods on simulated continuous control tasks with various action-cycle times. We show that our algorithm's performance is not affected by the choice of environment interaction frequency. Furthermore, we demonstrate the efficacy of CTCO in facilitating exploration in a real-world visual reaching task for a 7 DOF robotic arm with sparse rewards.