mlsanityData to 7 Oct 2026

Language as an Abstraction for Hierarchical Deep Reinforcement Learning

arXiv 1906.07343 ↗First posted 18 Jun 2019cs.LG cs.AI cs.CLcited by 138

Not on the map. Why?

Abstract (arXiv)

Solving complex, temporally-extended tasks is a long-standing problem in reinforcement learning (RL). We hypothesize that one critical element of solving such problems is the notion of compositionality. With the ability to learn concepts and sub-skills that can be composed to solve longer tasks, i.e. hierarchical RL, we can acquire temporally-extended behaviors. However, acquiring effective yet general abstractions for hierarchical RL is remarkably challenging. In this paper, we propose to use language as the abstraction, as it provides unique compositional structure, enabling fast learning and combinatorial generalization, while retaining tremendous flexibility, making it suitable for a variety of problems. Our approach learns an instruction-following low-level policy and a high-level policy that can reuse abstractions across tasks, in essence, permitting agents to reason using structured language. To study compositional task learning, we introduce an open-source object interaction environment built using the MuJoCo physics engine and the CLEVR engine. We find that, using our approach, agents can learn to solve to diverse, temporally-extended tasks such as object sorting and multi-object rearrangement, including from raw pixel observations. Our analysis reveals that the compositional nature of language is critical for learning diverse sub-skills and systematically generalizing to new sub-skills in comparison to non-compositional abstractions that use the same supervision.

Key facts. Cited by 138 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 2; 2021: 21; 2022: 42; 2023: 28; 2024: 25; 2025: 10; 2026: 10.

138 papers.

Sort:

By area:

  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  2. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  3. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  4. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  5. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  6. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  7. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  8. 2021Multi-Task Reinforcement Learning with Context-based Representationscited by 169
  9. 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 145
  10. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  11. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 114
  12. 2021Skill Induction and Planning with Latent Languagecited by 99
38 more
  1. 2023Modular Deep Learningcited by 82
  2. 2022Can Wikipedia Help Offline Reinforcement Learning?cited by 76
  3. 2021LTL2Action: Generalizing LTL Instructions for Multi-Task RLcited by 72
  4. 2024HiRT: Enhancing Robotic Control with Hierarchical Robot Transformerscited by 72
  5. 2021Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformerscited by 71
  6. 2023SkillDiffuser: Interpretable Hierarchical Planning via Skill Abstractions in Diffusion-Based Task Executioncited by 69
  7. 2024Scaling Instructable Agents Across Many Simulated Worldscited by 67
  8. 2024LLM-based Multi-Agent Reinforcement Learning: Current and Future Directionscited by 67
  9. 2023Learning to Model the World with Languagecited by 66
  10. 2022Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Modelscited by 58
  11. 2023DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignmentcited by 58
  12. 2022Improving Intrinsic Exploration with Language Abstractionscited by 49
  13. 2023LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programscited by 45
  14. 2021ELLA: Exploration through Learned Language Abstractioncited by 42
  15. 2022History Compression via Language Models in Reinforcement Learningcited by 39
  16. 2021Tell me why! Explanations support learning relational and causal structurecited by 36
  17. 2023Learning adaptive planning representations with natural language guidancecited by 36
  18. 2024BAGEL: Bootstrapping Agents by Guiding Exploration with Languagecited by 34
  19. 2026Dynamic Dual-Granularity Skill Bank for Agentic RLcited by 33
  20. 2023Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agentscited by 32
  21. 2022LISA: Learning Interpretable Skill Abstractions from Languagecited by 30
  22. 2023Thought Cloning: Learning to Think while Acting by Imitating Human Thinkingcited by 30
  23. 2021Interactive Learning from Activity Descriptioncited by 26
  24. 2022AnyMorph: Learning Transferable Polices By Inferring Agent Morphologycited by 25
  25. 2022Policy Architectures for Compositional Generalization in Controlcited by 23
  26. 2022EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RLcited by 23
  27. 2022How to talk so AI will learn: Instructions, descriptions, and autonomycited by 22
  28. 2022Learning Options via Compressioncited by 22
  29. 2025Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learningcited by 22
  30. 2022Intelligent problem-solving as integrated hierarchical reinforcement learningcited by 21
  31. 2022Generating Executable Action Plans with Environmentally-Aware Language Modelscited by 21
  32. 2020Inverse Reinforcement Learning with Natural Language Goalscited by 20
  33. 2022Language Control Diffusion: Efficiently Scaling through Space, Time, and Taskscited by 20
  34. 2021Solving Compositional Reinforcement Learning Problems via Task Reductioncited by 19
  35. 2024Compositional Automata Embeddings for Goal-Conditioned Reinforcement Learningcited by 18
  36. 2025MobileWorldBench: Towards Semantic World Modeling For Mobile Agentscited by 18
  37. 2023Semantic HELM: A Human-Readable Memory for Reinforcement Learningcited by 17
  38. 2023Structure in Deep Reinforcement Learning: A Survey and Open Problemscited by 16

50 most cited of 138 All 138 (JSON)