# Control Frequency Adaptation via Action Persistence in Batch Reinforcement Learning

arXiv: https://arxiv.org/abs/2002.06836 · HTML: https://mlsanity.com/paper/2002.06836/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Feb 2020.

## Built on by (most cited first, 28 of 28)

- [TempoRL: Learning When to Act](https://mlsanity.com/paper/2106.05262/) (2021)
- [Time Discretization-Invariant Safe Action Repetition for Policy Gradient Methods](https://mlsanity.com/paper/2111.03941/) (2021)
- [Delayed Reinforcement Learning by Imitation](https://mlsanity.com/paper/2205.05569/) (2022)
- [TAAC: Temporally Abstract Actor-Critic for Continuous Control](https://mlsanity.com/paper/2104.06521/) (2021)
- [Learning to Configure Separators in Branch-and-Cut](https://mlsanity.com/paper/2311.05650/) (2023)
- [Dynamic Decision Frequency with Continuous Options](https://mlsanity.com/paper/2212.04407/) (2022)
- [Addressing Action Oscillations through Learning Policy Inertia](https://mlsanity.com/paper/2103.02287/) (2021)
- [Temporal Transfer Learning for Traffic Optimization with Coarse-grained Advisory Autonomy](https://mlsanity.com/paper/2312.09436/) (2023)
- [No-Regret Reinforcement Learning in Smooth MDPs](https://mlsanity.com/paper/2402.03792/) (2024)
- [Towards Automatic Actor-Critic Solutions to Continuous Control](https://mlsanity.com/paper/2106.08918/) (2021)
- [PeRP: Personalized Residual Policies For Congestion Mitigation Through Co-operative Advisory Systems](https://mlsanity.com/paper/2308.00864/) (2023)
- [Local Linearity: the Key for No-regret Reinforcement Learning in Continuous MDPs](https://mlsanity.com/paper/2410.24071/) (2024)
- [Learning Uncertainty-Aware Temporally-Extended Actions](https://mlsanity.com/paper/2402.05439/) (2024)
- [Reinforcement Learning with Elastic Time Steps](https://mlsanity.com/paper/2402.14961/) (2024)
- [Simultaneously Updating All Persistence Values in Reinforcement Learning](https://mlsanity.com/paper/2211.11620/) (2022)
- [MOSEAC: Streamlined Variable Time Step Reinforcement Learning](https://mlsanity.com/paper/2406.01521/) (2024)
- [Managing Temporal Resolution in Continuous Value Estimation: A Fundamental Trade-off](https://mlsanity.com/paper/2212.08949/) (2022)
- [Actor-Critic with variable time discretization via sustained actions](https://mlsanity.com/paper/2308.04299/) (2023)
- [Performance Improvement Bounds for Lipschitz Configurable Markov Decision Processes](https://mlsanity.com/paper/2402.13821/) (2024)
- [State-Novelty Guided Action Persistence in Deep Reinforcement Learning](https://mlsanity.com/paper/2409.05433/) (2024)
- [TARC: Time-Adaptive Robotic Control](https://mlsanity.com/paper/2510.23176/) (2025)
- [When to Plan: Learning to Select Between Reactive Control and Deliberative Planning](https://mlsanity.com/paper/2607.16421/) (2026)
- [Learning Routines for Effective Off-Policy Reinforcement Learning](https://mlsanity.com/paper/2106.02943/) (2021)
- [Variable Time Step Reinforcement Learning for Robotic Applications](https://mlsanity.com/paper/2407.00290/) (2024)
- [Exploiting Risk-Aversion and Size-dependent fees in FX Trading with Fitted Natural Actor-Critic](https://mlsanity.com/paper/2410.23294/) (2024)
- [Inverse Delayed Reinforcement Learning](https://mlsanity.com/paper/2412.02931/) (2024)
- [Timescale Separation Enables Deep Reinforcement Learning Control of Rotating Detonation Engine Mode Transitions](https://mlsanity.com/paper/2604.14398/) (2026)
- [Latent Telepathy: Multi-Robot Communication with Self-Supervised Perceptual Latents](https://mlsanity.com/paper/2609.23269/) (2026)

## Abstract (arXiv)

The choice of the control frequency of a system has a relevant impact on the ability of reinforcement learning algorithms to learn a highly performing policy. In this paper, we introduce the notion of action persistence that consists in the repetition of an action for a fixed number of decision steps, having the effect of modifying the control frequency. We start analyzing how action persistence affects the performance of the optimal policy, and then we present a novel algorithm, Persistent Fitted Q-Iteration (PFQI), that extends FQI, with the goal of learning the optimal value function at a given persistence. After having provided a theoretical study of PFQI and a heuristic approach to identify the optimal persistence, we present an experimental campaign on benchmark domains to show the advantages of action persistence and proving the effectiveness of our persistence selection method.