mlsanityData to 7 Oct 2026

Learning Routines for Effective Off-Policy Reinforcement Learning

arXiv 2106.02943 ↗First posted 5 Jun 2021cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

The performance of reinforcement learning depends upon designing an appropriate action space, where the effect of each action is measurable, yet, granular enough to permit flexible behavior. So far, this process involved non-trivial user choices in terms of the available actions and their execution frequency. We propose a novel framework for reinforcement learning that effectively lifts such constraints. Within our framework, agents learn effective behavior over a routine space: a new, higher-level action space, where each routine represents a set of 'equivalent' sequences of granular actions with arbitrary length. Our routine space is learned end-to-end to facilitate the accomplishment of underlying off-policy reinforcement learning objectives. We apply our framework to two state-of-the-art off-policy algorithms and show that the resulting agents obtain relevant performance improvements while requiring fewer interactions with the environment per episode, improving computational efficiency.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 28 of its 47 references that are in the map. First posted on arXiv 5 Jun 2021.

Builds on

28 of 47 references.

References · 28

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  7. 2018DeepMind Control Suitecited by 935
  8. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  9. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  10. 2016The Option-Critic Architecturecited by 433
  11. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 328
  12. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 282
16 more
  1. 2019Challenges of Real-World Reinforcement Learningcited by 282
  2. 2016Variational Intrinsic Controlcited by 262
  3. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  4. 2015Deep Reinforcement Learning in Large Discrete Action Spacescited by 158
  5. 2017Stochastic Neural Networks for Hierarchical Reinforcement Learningcited by 124
  6. 2016A Deep Hierarchical Approach to Lifelong Learning in Minecraftcited by 90
  7. 2019Learning Action Representations for Reinforcement Learningcited by 75
  8. 2018Benchmarking Reinforcement Learning Algorithms on Real-World Robotscited by 64
  9. 2020Neural Dynamic Policies for End-to-End Sensorimotor Learningcited by 56
  10. 2018Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddingscited by 52
  11. 2020Continuous-Discrete Reinforcement Learning for Hybrid Control in Roboticscited by 42
  12. 2019Dynamics-aware Embeddingscited by 34
  13. 2016Strategic Attentive Writer for Learning Macro-Actionscited by 29
  14. 2020Temporally-Extended ε-Greedy Explorationcited by 29
  15. 2019The Natural Language of Actionscited by 28
  16. 2020Control Frequency Adaptation via Action Persistence in Batch Reinforcement Learningcited by 27

Built on by

Not cited yet.