An Analysis of Frame-skipping in Reinforcement Learning

In the practice of sequential decision making, agents are often designed to sense state at regular intervals of d time steps, d > 1, ignoring state information in between sensing steps. While it is clear that this practice can reduce sensing and compute costs, recent results indicate a further benefit. On many Atari console games, reinforcement learning (RL) algorithms deliver substantially better policies when run with d > 1 -- in fact with d even as high as 180. In this paper, we investigate the role of the parameter d in RL; d is called the "frame-skip" parameter, since states in the Atari domain are images. For evaluating a fixed policy, we observe that under standard conditions, frame-skipping does not affect asymptotic consistency. Depending on other parameters, it can possibly even benefit learning. To use d > 1 in the control setting, one must first specify which d-step open-loop action sequences can be executed in between sensing steps. We focus on "action-repetition", the common restriction of this choice to d-length sequences of the same action. We define a task-dependent quantity called the "price of inertia", in terms of which we upper-bound the loss incurred by action-repetition. We show that this loss may be offset by the gain brought to learning by a smaller task horizon. Our analysis is supported by experiments on different tasks and learning algorithms.