mlsanityData to 14 Oct 2026

Utilizing Skipped Frames in Action Repeats via Pseudo-Actions

arXiv 2105.03041 ↗First posted 7 May 2021cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

In many deep reinforcement learning settings, when an agent takes an action, it repeats the same action a predefined number of times without observing the states until the next action-decision point. This technique of action repetition has several merits in training the agent, but the data between action-decision points (i.e., intermediate frames) are, in effect, discarded. Since the amount of training data is inversely proportional to the interval of action repeats, they can have a negative impact on the sample efficiency of training. In this paper, we propose a simple but effective approach to alleviate to this problem by introducing the concept of pseudo-actions. The key idea of our method is making the transition between action-decision points usable as training data by considering pseudo-actions. Pseudo-actions for continuous control tasks are obtained as the average of the action sequence straddling an action-decision point. For discrete control tasks, pseudo-actions are computed from learned action embeddings. This method can be combined with any model-free reinforcement learning algorithm that involves the learning of Q-functions. We demonstrate the effectiveness of our approach on both continuous and discrete control tasks in OpenAI Gym.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 23 of its 28 references that are in the map. First posted on arXiv 7 May 2021.

Builds on

23 of 28 references.

References · 23

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2016Layer Normalizationcited by 5,669
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  6. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  7. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  8. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  9. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  10. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  11. 2018DeepMind Control Suitecited by 956
  12. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
11 more
  1. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  2. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  3. 2019Model-Based Reinforcement Learning for Ataricited by 502
  4. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 274
  5. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  6. 2017Noisy Networks for Explorationcited by 266
  7. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 231
  8. 2018Multi-task Deep Reinforcement Learning with PopArtcited by 141
  9. 2019Learning Action Representations for Reinforcement Learningcited by 75
  10. 2015Increasing the Action Gap: New Operators for Reinforcement Learningcited by 49
  11. 2019On Inductive Biases in Deep Reinforcement Learningcited by 24

Built on by

Not cited yet.