# An Analysis of Frame-skipping in Reinforcement Learning

arXiv: https://arxiv.org/abs/2102.03718 · HTML: https://mlsanity.com/paper/2102.03718/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 38 references that are in the map.
- First posted on arXiv 7 Feb 2021.

## Builds on (most cited first, 6 of 6)

- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Learning to Repeat: Fine Grained Action Repetition for Deep Reinforcement Learning](https://mlsanity.com/paper/1702.06054/) (2017)
- [True Online Temporal-Difference Learning](https://mlsanity.com/paper/1512.04087/) (2015)
- [Deep Reinforcement Learning With Macro-Actions](https://mlsanity.com/paper/1606.04615/) (2016)
- [Dynamic Frame skip Deep Q Network](https://mlsanity.com/paper/1605.05365/) (2016)
- [Constructing Abstraction Hierarchies Using a Skill-Symbol Loop](https://mlsanity.com/paper/1509.07582/) (2015)

## Built on by (most cited first, 18 of 18)

- [Learning When to Plan: Efficiently Allocating Test-Time Compute for LLM Agents](https://mlsanity.com/paper/2509.03581/) (2025)
- [Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returns](https://mlsanity.com/paper/2503.03660/) (2025)
- [Optimization of the Model Predictive Control Meta-Parameters Through Reinforcement Learning](https://mlsanity.com/paper/2111.04146/) (2021)
- [Towards Automatic Actor-Critic Solutions to Continuous Control](https://mlsanity.com/paper/2106.08918/) (2021)
- [The RL Perceptron: Generalisation Dynamics of Policy Learning in High Dimensions](https://mlsanity.com/paper/2306.10404/) (2023)
- [Simultaneously Updating All Persistence Values in Reinforcement Learning](https://mlsanity.com/paper/2211.11620/) (2022)
- [Optimizing Attention and Cognitive Control Costs Using Temporally-Layered Architectures](https://mlsanity.com/paper/2305.18701/) (2023)
- [Speeding-up Evolutionary Algorithms to solve Black-Box Optimization Problems](https://mlsanity.com/paper/2309.13349/) (2023)
- [Bounding Distributional Shifts in World Modeling through Novelty Detection](https://mlsanity.com/paper/2508.06096/) (2025)
- [Temporally Layered Architecture for Adaptive, Distributed and Continuous Control](https://mlsanity.com/paper/2301.00723/) (2022)
- [Actor-Critic with variable time discretization via sustained actions](https://mlsanity.com/paper/2308.04299/) (2023)
- [For How Long Should We Be Punching? Learning Action Duration in Fighting Games](https://mlsanity.com/paper/2605.20911/) (2026)
- [Hybrid Reinforcement Learning for Optimizing Pump Sustainability in Real-World Water Distribution Networks](https://mlsanity.com/paper/2310.09412/) (2023)
- [Planning the path with Reinforcement Learning: Optimal Robot Motion Planning in RoboCup Small Size League Environments](https://mlsanity.com/paper/2404.15410/) (2024)
- [Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control](https://mlsanity.com/paper/2410.08979/) (2024)
- [Select before Act: Spatially Decoupled Action Repetition for Continuous Control](https://mlsanity.com/paper/2502.06919/) (2025)
- [LLMs Are Not Good Strategists, Yet Memory-Enhanced Agency Boosts Reasoning](https://mlsanity.com/paper/2608.12626/) (2026)
- [Separating Decision Time from Decision Quality in the Real-Time Gap of Distilled Deciders: Evidence from a Game and a Conveyor Simulator](https://mlsanity.com/paper/2610.04810/) (2026)

## Abstract (arXiv)

In the practice of sequential decision making, agents are often designed to sense state at regular intervals of d time steps, d > 1, ignoring state information in between sensing steps. While it is clear that this practice can reduce sensing and compute costs, recent results indicate a further benefit. On many Atari console games, reinforcement learning (RL) algorithms deliver substantially better policies when run with d > 1 -- in fact with d even as high as 180. In this paper, we investigate the role of the parameter d in RL; d is called the "frame-skip" parameter, since states in the Atari domain are images. For evaluating a fixed policy, we observe that under standard conditions, frame-skipping does not affect asymptotic consistency. Depending on other parameters, it can possibly even benefit learning. To use d > 1 in the control setting, one must first specify which d-step open-loop action sequences can be executed in between sensing steps. We focus on "action-repetition", the common restriction of this choice to d-length sequences of the same action. We define a task-dependent quantity called the "price of inertia", in terms of which we upper-bound the loss incurred by action-repetition. We show that this loss may be offset by the gain brought to learning by a smaller task horizon. Our analysis is supported by experiments on different tasks and learning algorithms.