mlsanityData to 7 Oct 2026

Dynamic Decision Frequency with Continuous Options

arXiv 2212.04407 ↗First posted 6 Dec 2022cs.LG cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

In classic reinforcement learning algorithms, agents make decisions at discrete and fixed time intervals. The duration between decisions becomes a crucial hyperparameter, as setting it too short may increase the problem's difficulty by requiring the agent to make numerous decisions to achieve its goal while setting it too long can result in the agent losing control over the system. However, physical systems do not necessarily require a constant control frequency, and for learning agents, it is often preferable to operate with a low frequency when possible and a high frequency when necessary. We propose a framework called Continuous-Time Continuous-Options (CTCO), where the agent chooses options as sub-policies of variable durations. These options are time-continuous and can interact with the system at any desired frequency providing a smooth change of actions. We demonstrate the effectiveness of CTCO by comparing its performance to classical RL and temporal-abstraction RL methods on simulated continuous control tasks with various action-cycle times. We show that our algorithm's performance is not affected by the choice of environment interaction frequency. Furthermore, we demonstrate the efficacy of CTCO in facilitating exploration in a real-world visual reaching task for a 7 DOF robotic arm with sparse rewards.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 16 of its 26 references that are in the map. First posted on arXiv 6 Dec 2022.

Builds on

16 of 26 references.

References · 16

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2018DeepMind Control Suitecited by 935
  4. 2016The Option-Critic Architecturecited by 433
  5. 2017When Waiting is not an Option : Learning Options with a Deliberation Costcited by 57
  6. 2019Making Deep Q-learning methods robust to time discretizationcited by 51
  7. 2019DAC: The Double Actor-Critic Architecture for Learning Optionscited by 43
  8. 2016Unifying task specification in reinforcement learningcited by 41
  9. 2021Continuous-Time Model-Based Reinforcement Learningcited by 39
  10. 2017Learning to Repeat: Fine Grained Action Repetition for Deep Reinforcement Learningcited by 38
  11. 2018Setting up a Reinforcement Learning Task with a Real-World Robotcited by 32
  12. 2020Thinking While Moving: Deep Reinforcement Learning with Concurrent Controlcited by 29
4 more
  1. 2020Control Frequency Adaptation via Action Persistence in Batch Reinforcement Learningcited by 27
  2. 2021Time Discretization-Invariant Safe Action Repetition for Policy Gradient Methodscited by 21
  3. 2022Asynchronous Reinforcement Learning for Real-Time Control of Physical Robotscited by 19
  4. 2020Locally Persistent Exploration in Continuous Control Tasks with Sparse Rewardscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 3; 2026: 3.

11 papers.

  1. 2024Revisiting Sparse Rewards for Goal-Reaching Reinforcement Learningcited by 12
  2. 2024Enabling Realtime Reinforcement Learning at Scale with Staggered Asynchronous Inferencecited by 7
  3. 2024Reinforcement Learning with Elastic Time Stepscited by 2
  4. 2024MOSEAC: Streamlined Variable Time Step Reinforcement Learningcited by 2
  5. 2024An Idiosyncrasy of Time-discretization in Reinforcement Learningcited by 2
  6. 2026When to Plan: Learning to Select Between Reactive Control and Deliberative Planningcited by 1
  7. 2025A Robust Model-Based Approach for Continuous-Time Policy Evaluation with Unknown L'evy Process Dynamicscited by 0
  8. 2025Learning Multi-Timescale Interventions under Safety and Resource Constraintscited by 0
  9. 2025TARC: Time-Adaptive Robotic Controlcited by 0
  10. 2026Timescale Separation Enables Deep Reinforcement Learning Control of Rotating Detonation Engine Mode Transitionscited by 0
  11. 2026Action Chunking Proximal Policy Optimization with Feedback Correctioncited by 0