mlsanityData to 7 Oct 2026

When to Sense and Control? A Time-adaptive Approach for Continuous-Time RL

arXiv 2406.01163 ↗First posted 3 Jun 2024cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning (RL) excels in optimizing policies for discrete-time Markov decision processes (MDP). However, various systems are inherently continuous in time, making discrete-time MDPs an inexact modeling choice. In many applications, such as greenhouse control or medical treatments, each interaction (measurement or switching of action) involves manual intervention and thus is inherently costly. Therefore, we generally prefer a time-adaptive approach with fewer interactions with the system. In this work, we formalize an RL framework, Time-adaptive Control & Sensing (TaCoS), that tackles this challenge by optimizing over policies that besides control predict the duration of its application. Our formulation results in an extended MDP that any standard RL algorithm can solve. We demonstrate that state-of-the-art RL algorithms trained on TaCoS drastically reduce the interaction amount over their discrete-time counterpart while retaining the same or improved performance, and exhibiting robustness over discretization frequency. Finally, we propose OTaCoS, an efficient model-based algorithm for our setting. We show that OTaCoS enjoys sublinear regret for systems with sufficiently smooth dynamics and empirically results in further sample-efficiency gains.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 35 of its 68 references that are in the map. First posted on arXiv 3 Jun 2024.

Builds on

35 of 68 references.

References · 35

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2018Neural Ordinary Differential Equationscited by 3,733
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  6. 2015Trust Region Policy Optimizationcited by 2,503
  7. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  8. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  9. 2019Hamiltonian Neural Networkscited by 702
  10. 2020Lagrangian Neural Networkscited by 388
  11. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
  12. 2020On Information Gain and Regret Bounds in Gaussian Process Banditscited by 156
23 more
  1. 2020Information Theoretic Regret Bounds for Online Nonlinear Controlcited by 87
  2. 2020Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planningcited by 83
  3. 2018Information Directed Sampling and Bandits with Heteroscedastic Noisecited by 75
  4. 2017When Waiting is not an Option : Learning Options with a Deliberation Costcited by 57
  5. 2021Continuous-Time Model-Based Reinforcement Learningcited by 39
  6. 2017Learning to Repeat: Fine Grained Action Repetition for Deep Reinforcement Learningcited by 38
  7. 2023Optimistic Active Exploration of Dynamical Systemscited by 32
  8. 2016Strategic Attentive Writer for Learning Macro-Actionscited by 29
  9. 2020Temporally-Extended ε-Greedy Explorationcited by 29
  10. 2019AMZ Driverless: The Full Autonomous Racing Systemcited by 28
  11. 2022Semi-Supervised Offline Reinforcement Learning with Action-Free Trajectoriescited by 27
  12. 2021TempoRL: Learning When to Actcited by 25
  13. 2023Efficient Exploration in Continuous-time Model-based Reinforcement Learningcited by 21
  14. 2021Value Iteration in Continuous Actions, States and Timecited by 17
  15. 2021TAAC: Temporally Abstract Actor-Critic for Continuous Controlcited by 14
  16. 2016Deep Reinforcement Learning With Macro-Actionscited by 13
  17. 2023Hallucinated Adversarial Control for Conservative Offline Policy Evaluationcited by 13
  18. 2023Neural Markov Jump Processescited by 11
  19. 2023Act-Then-Measure: Reinforcement Learning for Partially Observable Environments with Active Measuringcited by 10
  20. 2021Addressing Action Oscillations through Learning Policy Inertiacited by 8
  21. 2019Continuous-Time Markov Decision Processes with Controlled Observationscited by 7
  22. 2021Towards Automatic Actor-Critic Solutions to Continuous Controlcited by 4
  23. 2022Offline Reinforcement Learning at Multiple Frequenciescited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 3.

9 papers.

  1. 2025Toward Agents That Reason About Their Computationcited by 3
  2. 2025Sample-efficient and Scalable Exploration in Continuous-Time RLcited by 3
  3. 2025Sample and Computationally Efficient Continuous-Time Reinforcement Learning with General Function Approximationcited by 2
  4. 2025MDPs with a State Sensing Costcited by 1
  5. 2025Instance-Dependent Continuous-Time Reinforcement Learning via Maximum Likelihood Estimationcited by 1
  6. 2026Learning When to Act: Communication-Efficient Reinforcement Learning via Run-Time Assurancecited by 1
  7. 2025TARC: Time-Adaptive Robotic Controlcited by 0
  8. 2026A Unified Benchmark for Dynamic Medical Treatment Reinforcement Learningcited by 0
  9. 2026Provably Efficient Reinforcement Learning in Continuous-Time Episodic MDPs with Poisson Decision Epochscited by 0