mlsanityData to 7 Oct 2026

DAC: The Double Actor-Critic Architecture for Learning Options

arXiv 1904.12691 ↗First posted 29 Apr 2019cs.LG cs.AI stat.MLcited by 43

Not on the map. Why?

Abstract (arXiv)

We reformulate the option framework as two parallel augmented MDPs. Under this novel formulation, all policy optimization algorithms can be used off the shelf to learn intra-option policies, option termination conditions, and a master policy over options. We apply an actor-critic algorithm on each augmented MDP, yielding the Double Actor-Critic (DAC) architecture. Furthermore, we show that, when state-value functions are used as critics, one critic can be expressed in terms of the other, and hence only one critic is necessary. We conduct an empirical study on challenging robot simulation tasks. In a transfer learning setting, DAC outperforms both its hierarchy-free counterpart and previous gradient-based option learning algorithms.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 11; 2023: 5; 2024: 7; 2025: 6; 2026: 6.

43 papers.

Sort:

By area:

  1. 2021Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Taskscited by 75
  2. 2022On the Robustness of Safe Reinforcement Learning under Observational Perturbationscited by 38
  3. 2022LEAGUE: Guided Skill Learning and Abstraction for Long-Horizon Manipulationcited by 32
  4. 2021Flexible Option Learningcited by 24
  5. 2021Efficient Continuous Control with Double Actors and Regularized Criticscited by 20
  6. 2021Adversarial Option-Aware Hierarchical Imitation Learningcited by 19
  7. 2022Option-Aware Adversarial Inverse Reinforcement Learning for Robotic Controlcited by 16
  8. 2023Multi-task Hierarchical Adversarial Inverse Reinforcement Learningcited by 15
  9. 2026HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agentscited by 14
  10. 2022Plan Your Target and Learn Your Skills: Transferable State-Only Imitation Learning via Decoupled Policy Optimizationcited by 12
  11. 2022Dynamic Decision Frequency with Continuous Optionscited by 11
  12. 2022MO2: Model-Based Offline Optionscited by 10
31 more
  1. 2021Measuring Progress in Deep Reinforcement Learning Sample Efficiencycited by 8
  2. 2021Program Synthesis Guided Reinforcement Learning for Partially Observed Environmentscited by 7
  3. 2022A Unified Algorithm Framework for Unsupervised Discovery of Skills based on Determinantal Point Processcited by 7
  4. 2024A Simple Mixture Policy Parameterization for Improving Sample Efficiency of CVaR Optimizationcited by 7
  5. 2022Autonomous Unmanned Aerial Vehicle Navigation using Reinforcement Learning: A Systematic Reviewcited by 6
  6. 2023Skill-Critic: Refining Learned Skills for Hierarchical Reinforcement Learningcited by 6
  7. 2024IDIL: Imitation Learning of Intent-Driven Expert Behaviorcited by 5
  8. 2023Hierarchical Deep Counterfactual Regret Minimizationcited by 4
  9. 2022A Learning Approach for Joint Design of Event-triggered Control and Power-Efficient Resource Allocationcited by 2
  10. 2024Reinforcement Learning via Auxiliary Task Distillationcited by 2
  11. 2026Quantum Hierarchical Reinforcement Learning via Variational Quantum Circuitscited by 2
  12. 2024SOAP-RL: Sequential Option Advantage Propagation for Reinforcement Learning in POMDP Environmentscited by 1
  13. 2024Subgoal Discovery Using a Free Energy Paradigm and State Aggregationscited by 1
  14. 2025Optimizing Electric Bus Charging Scheduling with Uncertainties Using Hierarchical Deep Reinforcement Learningcited by 1
  15. 2026Zero-shot Imitation Learning by Latent Topology Mappingcited by 1
  16. 2021Interpretable Reinforcement Learning Inspired by Piaget's Theory of Cognitive Developmentcited by 0
  17. 2021Auto-COP: Adaptation Generation in Context-Oriented Programming using Reinforcement Learning Optionscited by 0
  18. 2022Planning Immediate Landmarks of Targets for Model-Free Skill Transfer across Agentscited by 0
  19. 2022Reusable Options through Gradient-based Meta Learningcited by 0
  20. 2023Actor-Director-Critic: A Novel Deep Reinforcement Learning Frameworkcited by 0
  21. 2023Wasserstein Diversity-Enriched Regularizer for Hierarchical Reinforcement Learningcited by 0
  22. 2024Bidirectional-Reachable Hierarchical Reinforcement Learning with Mutually Responsive Policiescited by 0
  23. 2024Accelerating Task Generalisation with Multi-Level Skill Hierarchiescited by 0
  24. 2025Multi-Timescale Hierarchical Reinforcement Learning for Unified Behavior and Control of Autonomous Drivingcited by 0
  25. 2025Learning Temporal Abstractions via Variational Homomorphisms in Option-Induced Abstract MDPscited by 0
  26. 2025Learning to Drive Safely with Hybrid Optionscited by 0
  27. 2025Convergence and stability of Q-learning in Hierarchical Reinforcement Learningcited by 0
  28. 2025Safe and Sustainable Electric Bus Charging Scheduling with Constrained Hierarchical DRLcited by 0
  29. 2026Hierarchical Reward Design from Language: Enhancing Alignment of Agent Behavior with Human Specificationscited by 0
  30. 2026ToSCA: Leveraging Hierarchical Reinforcement Learning on Temporal and Strategic Abstractions of Conversational Agentscited by 0
  31. 2026When Do Options Help? Policy Necrosis and Redundant Coverage in Option-Criticcited by 0