mlsanityData to 14 Oct 2026

The Option-Critic Architecture

arXiv 1609.05140 ↗First posted 16 Sep 2016cs.AIcited by 448

Not on the map. Why?

Cited by 114 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Temporal abstraction is key to scaling up learning and planning in reinforcement learning. While planning with temporally extended actions is well understood, creating such abstractions autonomously from data has remained challenging. We tackle this problem in the framework of options [Sutton, Precup & Singh, 1999; Precup, 2000]. We derive policy gradient theorems for options and propose a new option-critic architecture capable of learning both the internal policies and the termination conditions of options, in tandem with the policy over options, and without the need to provide any additional rewards or subgoals. Experimental results in both discrete and continuous environments showcase the flexibility and efficiency of the framework.

Key facts. Cited by 448 papers in the map (data through 14 Oct 2026). First posted on arXiv 16 Sep 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2018: 2; 2019: 1; 2020: 4; 2021: 74; 2022: 76; 2023: 60; 2024: 60; 2025: 78; 2026: 93.

448 papers.

Sort:

By area:

  1. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 398
  2. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 228
  3. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 166
  4. 2023Structured World Models from Human Videoscited by 147
  5. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 141
  6. 2024HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulationcited by 112
  7. 2025Reinforcement Learning with Action Chunkingcited by 107
  8. 2021Skill Induction and Planning with Latent Languagecited by 99
  9. 2023Generative Skill Chaining: Long-Horizon Skill Planning with Diffusion Modelscited by 91
  10. 2021From Motor Control to Team Play in Simulated Humanoid Footballcited by 87
  11. 2023Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidancecited by 86
  12. 2023Modular Deep Learningcited by 83
38 more
  1. 2022Latent Plans for Task-Agnostic Offline Reinforcement Learningcited by 81
  2. 2024Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systemscited by 81
  3. 2022BITS: Bi-level Imitation for Traffic Simulationcited by 80
  4. 2021Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Taskscited by 78
  5. 2021Explainable Artificial Intelligence for Autonomous Driving: A Comprehensive Overview and Field Guide for Future Research Directionscited by 73
  6. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 70
  7. 2021The Option Keyboard: Combining Skills in Reinforcement Learningcited by 67
  8. 2021Accelerating Robotic Reinforcement Learning via Parameterized Action Primitivescited by 65
  9. 2023Multi-Stage Cable Routing through Hierarchical Imitation Learningcited by 60
  10. 2021Hierarchical Reinforcement Learning By Discovering Intrinsic Optionscited by 58
  11. 2022Learning Neuro-Symbolic Skills for Bilevel Planningcited by 58
  12. 2022Efficient Planning in a Compact Latent Action Spacecited by 57
  13. 2025Horizon Reduction Makes RL Scalablecited by 57
  14. 2021Temporal Abstraction in Reinforcement Learning with the Successor Representationcited by 54
  15. 2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradientcited by 51
  16. 2023PlayFusion: Skill Acquisition via Diffusion from Language-Annotated Playcited by 50
  17. 2023Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulationcited by 49
  18. 2023Lyfe Agents: Generative agents for low-cost real-time social interactionscited by 49
  19. 2021Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularizationcited by 46
  20. 2021Same State, Different Task: Continual Reinforcement Learning without Interferencecited by 43
  21. 2026Hierarchical Planning with Latent World Modelscited by 43
  22. 2022Modular Lifelong Reinforcement Learning via Neural Compositioncited by 40
  23. 2021TRAIL: Near-Optimal Imitation Learning with Suboptimal Datacited by 39
  24. 2024Code as Reward: Empowering Reinforcement Learning with VLMscited by 39
  25. 2021Hierarchical Skills for Efficient Explorationcited by 38
  26. 2023Efficient Reinforcement Learning for Autonomous Driving with Parameterized Skills and Priorscited by 36
  27. 2022LEAGUE: Guided Skill Learning and Abstraction for Long-Horizon Manipulationcited by 34
  28. 2021LASER: Learning a Latent Action Space for Efficient Reinforcement Learningcited by 32
  29. 2024Position: Theory of Mind Benchmarks are Broken for Large Language Modelscited by 31
  30. 2021Coach-Player Multi-Agent Reinforcement Learning for Dynamic Team Compositioncited by 30
  31. 2022A Survey on Recent Advances and Challenges in Reinforcement Learning Methods for Task-Oriented Dialogue Policy Learningcited by 30
  32. 2023Deep Laplacian-based Options for Temporally-Extended Explorationcited by 29
  33. 2022An information-theoretic perspective on intrinsic motivation in reinforcement learning: a surveycited by 28
  34. 2021Discovery of Options via Meta-Learned Subgoalscited by 27
  35. 2021Probabilistic Mixture-of-Experts for Efficient Deep Reinforcement Learningcited by 27
  36. 2022How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Compositioncited by 27
  37. 2021Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoningcited by 26
  38. 2021Learning Transferable Motor Skills with Hierarchical Latent Mixture Policiescited by 26

50 most cited of 448 All 448 (JSON)