Variable Time Step Reinforcement Learning for Robotic Applications
On the map: Deep reinforcement learning methods → RL for robot locomotion and manipulation
Abstract (arXiv)
Traditional reinforcement learning (RL) generates discrete control policies, assigning one action per cycle. These policies are usually implemented as in a fixed-frequency control loop. This rigidity presents challenges as optimal control frequency is task-dependent; suboptimal frequencies increase computational demands and reduce exploration efficiency. Variable Time Step Reinforcement Learning (VTS-RL) addresses these issues with adaptive control frequencies, executing actions only when necessary, thus reducing computational load and extending the action space to include action durations. In this paper we introduce the Multi-Objective Soft Elastic Actor-Critic (MOSEAC) method to perform VTS-RL, validating it through theoretical analysis and experimentation in simulation and on real robots. Results show faster convergence, better training results, and reduced energy consumption with respect to other variable- or fixed-frequency approaches.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods. Builds on 21 of its 54 references that are in the map. First posted on arXiv 29 Jun 2024.
Builds on
21 of 54 references.
References · 21
- 2017Attention Is All You Needcited by 60,361
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
- 2015End-to-End Training of Deep Visuomotor Policiescited by 875
- 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
- 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 213
- 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 167
- 2020Reinforcement Learning with Random Delayscited by 59
- 2019Hierarchical Reinforcement Learning with Advantage-Based Auxiliary Rewardscited by 40
- 2017Learning to Repeat: Fine Grained Action Repetition for Deep Reinforcement Learningcited by 39
- 2021Deep Reinforcement Learning for the Control of Robotic Manipulation: A Focussed Mini-Reviewcited by 36
- 2019DOOR-SLAM: Distributed, Online, and Outlier Resilient SLAM for Robotic Teamscited by 29
9 more
- 2020Control Frequency Adaptation via Action Persistence in Batch Reinforcement Learningcited by 28
- 2021Time Discretization-Invariant Safe Action Repetition for Policy Gradient Methodscited by 23
- 2020Reinforcement Learning Approaches in Social Roboticscited by 16
- 2020Locally Persistent Exploration in Continuous Control Tasks with Sparse Rewardscited by 13
- 2022Dynamic Decision Frequency with Continuous Optionscited by 13
- 2024To the Max: Reinventing Reward in Reinforcement Learningcited by 8
- 2024Assessing the Impact of Distribution Shift on Reinforcement Learning Performancecited by 7
- 2024Deployable Reinforcement Learning with Variable Control Ratecited by 5
- 2024Reinforcement Learning with Elastic Time Stepscited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024MOSEAC: Streamlined Variable Time Step Reinforcement Learning
- 2025Taxonomy and Trends in Reinforcement Learning for Robotics and Control Systems: A Structured Review
- 2024Reinforcement Learning Controllers for Soft Robots using Learned Environments
- 2026Physics-Informed Policy Optimization via Analytic Dynamics Regularization
- 2024SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learning
- 2026Foundations of Reinforcement Learning and Control:Connections and New Perspectives
- 2025SAC-MoE: Reinforcement Learning with Mixture-of-Experts for Control of Hybrid Dynamical Systems with Uncertainty
- 2026Actuator Dynamics Curricula for Narrow-Viability Tasks in Legged Robot Learning