mlsanityData to 7 Oct 2026

TAAC: Temporally Abstract Actor-Critic for Continuous Control

arXiv 2104.06521 ↗First posted 13 Apr 2021cs.LG cs.AI cs.ROcited by 14

Not on the map. Why?

Abstract (arXiv)

We present temporally abstract actor-critic (TAAC), a simple but effective off-policy RL algorithm that incorporates closed-loop temporal abstraction into the actor-critic framework. TAAC adds a second-stage binary policy to choose between the previous action and a new action output by an actor. Crucially, its "act-or-repeat" decision hinges on the actually sampled action instead of the expected behavior of the actor. This post-acting switching scheme let the overall policy make more informed decisions. TAAC has two important features: a) persistent exploration, and b) a new compare-through Q operator for multi-step TD backup, specially tailored to the action repetition scenario. We demonstrate TAAC's advantages over several strong baselines across 14 continuous control tasks. Our surprising finding reveals that while achieving top performance, TAAC is able to "mine" a significant number of repeated actions with the trained policy even on continuous tasks whose problem structures on the surface seem to repel action repetition. This suggests that aside from encouraging persistent exploration, action repetition can find its place in a good policy behavior. Code is available at https://github.com/hnyu/taac.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 28 of its 56 references that are in the map. First posted on arXiv 13 Apr 2021.

Builds on

28 of 56 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  7. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  8. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  9. 2016OpenAI Gymcited by 1,684
  10. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  11. 2017Hindsight Experience Replaycited by 1,096
  12. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
16 more
  1. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
  2. 2016The Option-Critic Architecturecited by 433
  3. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  4. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 328
  5. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  6. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
  7. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 164
  8. 2021Hierarchical Reinforcement Learning By Discovering Intrinsic Optionscited by 57
  9. 2019Sub-policy Adaptation for Hierarchical Reinforcement Learningcited by 47
  10. 2020Continuous-Discrete Reinforcement Learning for Hybrid Control in Roboticscited by 42
  11. 2019Discrete and Continuous Action Representation for Practical RL in Video Gamescited by 34
  12. 2018Relative Entropy Regularized Policy Iterationcited by 32
  13. 2020Temporally-Extended ε-Greedy Explorationcited by 29
  14. 2020Control Frequency Adaptation via Action Persistence in Batch Reinforcement Learningcited by 27
  15. 2020Locally Persistent Exploration in Continuous Control Tasks with Sparse Rewardscited by 12
  16. 2021Addressing Action Oscillations through Learning Policy Inertiacited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 4; 2024: 3; 2025: 3; 2026: 1.

14 papers.

Sort:

By area:

  1. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 103
  2. 2024Open the Black Box: Step-based Policy Updates for Temporally-Correlated Episodic Reinforcement Learningcited by 12
  3. 2022Generative Planning for Temporally Coordinated Exploration in Reinforcement Learningcited by 10
  4. 2024When to Sense and Control? A Time-adaptive Approach for Continuous-Time RLcited by 9
  5. 2023LESSON: Learning to Integrate Exploration Strategies for Reinforcement Learning via an Option Frameworkcited by 7
  6. 2022Simultaneously Updating All Persistence Values in Reinforcement Learningcited by 2
  7. 2023Optimizing Attention and Cognitive Control Costs Using Temporally-Layered Architecturescited by 2
  8. 2022Temporally Layered Architecture for Adaptive, Distributed and Continuous Controlcited by 1
  9. 2025Action Robust Reinforcement Learning via Optimal Adversary Aware Policy Optimizationcited by 1
  10. 2023Actor-Director-Critic: A Novel Deep Reinforcement Learning Frameworkcited by 0
  11. 2024Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Controlcited by 0
  12. 2025Select before Act: Spatially Decoupled Action Repetition for Continuous Controlcited by 0
2 more
  1. 2025Deep Reinforcement Learning for Day-to-day Dynamic Tolling in Tradable Credit Schemescited by 0
  2. 2026Interaction-Limited Safe Continuous-Time RL for Dynamical Medical Treatmentcited by 0