mlsanityData to 7 Oct 2026

Skill Induction and Planning with Latent Language

arXiv 2110.01517 ↗First posted 4 Oct 2021cs.LG cs.AI cs.CLcited by 99

Not on the map. Why?

Abstract (arXiv)

We present a framework for learning hierarchical policies from demonstrations, using sparse natural language annotations to guide the discovery of reusable skills for autonomous decision-making. We formulate a generative model of action sequences in which goals generate sequences of high-level subtask descriptions, and these descriptions generate sequences of low-level actions. We describe how to train this model using primarily unannotated demonstrations by parsing demonstrations into sequences of named high-level subtasks, using only a small number of seed annotations to ground language in action. In trained models, natural language commands index a combinatorial library of skills; agents can use these skills to plan by generating high-level instruction sequences tailored to novel goals. We evaluate this approach in the ALFRED household simulation environment, providing natural language annotations for only 10% of demonstrations. It achieves task completion rates comparable to state-of-the-art models (outperforming several recent methods with access to ground-truth plans during training and evaluation) while providing structured and human-readable high-level plans.

Key facts. Cited by 99 papers in the map (data through 7 Oct 2026). Builds on 28 of its 59 references that are in the map. First posted on arXiv 4 Oct 2021.

Builds on

28 of 59 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  5. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  6. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  7. 2016The Option-Critic Architecturecited by 433
  8. 2015Neural Module Networkscited by 375
  9. 2017Embodied Question Answeringcited by 335
  10. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 282
  11. 2016Modular Multitask Reinforcement Learning with Policy Sketchescited by 200
  12. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
16 more
  1. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  2. 2017IQA: Visual Question Answering in Interactive Environmentscited by 154
  3. 2019Language as an Abstraction for Hierarchical Deep Reinforcement Learningcited by 138
  4. 2021A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Executioncited by 91
  5. 2017Mapping Instructions and Visual Observations to Actions with Reinforcement Learningcited by 80
  6. 2017Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learningcited by 77
  7. 2017Learning with Latent Languagecited by 75
  8. 2018TACO: Learning Task Decomposition via Temporal Alignment for Controlcited by 61
  9. 2018Learning Neural Templates for Text Generationcited by 61
  10. 2021Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completioncited by 57
  11. 2021Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoringcited by 53
  12. 2017Multi-Level Discovery of Deep Optionscited by 50
  13. 2021PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D Worldcited by 47
  14. 2021Leveraging Language to Learn Program Abstractions and Search Heuristicscited by 44
  15. 2019Hierarchical Decision Making by Generating and Following Natural Language Instructionscited by 30
  16. 2021Multitasking Inhibits Semantic Driftcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 17; 2023: 34; 2024: 26; 2025: 14; 2026: 7.

99 papers.

Sort:

By area:

  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  2. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  3. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  4. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  5. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  6. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  7. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  8. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  9. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 231
  10. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  11. 2024RT-H: Action Hierarchies Using Languagecited by 204
  12. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
38 more
  1. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  2. 2023ADaPT: As-Needed Decomposition and Planning with Language Modelscited by 172
  3. 2023Generalized Planning in PDDL Domains with Pretrained Large Language Modelscited by 139
  4. 2023ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigationcited by 126
  5. 2024Yell At Your Robot: Improving On-the-Fly from Language Correctionscited by 123
  6. 2025Inducing Programmatic Skills for Agentic Taskscited by 103
  7. 2023Compositional Foundation Models for Hierarchical Planningcited by 92
  8. 2023Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Taskcited by 91
  9. 2025PlanGenLLMs: A Modern Survey of LLM Planning Capabilitiescited by 82
  10. 2023ISR-LLM: Iterative Self-Refined Large Language Model for Long-Horizon Sequential Task Planningcited by 77
  11. 2023Eight Things to Know about Large Language Modelscited by 73
  12. 2023Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agentscited by 66
  13. 2023OMNI: Open-endedness via Models of human Notions of Interestingnesscited by 66
  14. 2023Learning to Model the World with Languagecited by 66
  15. 2023Grammar Prompting for Domain-Specific Language Generation with Large Language Modelscited by 58
  16. 2022Structured, flexible, and robust: benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning taskscited by 56
  17. 2022Learning Neuro-Symbolic Skills for Bilevel Planningcited by 55
  18. 2023LangNav: Language as a Perceptual Representation for Navigationcited by 47
  19. 2024COMBO: Compositional World Models for Embodied Multi-Agent Cooperationcited by 47
  20. 2023LILO: Learning Interpretable Libraries by Compressing and Documenting Codecited by 46
  21. 2024Robot Utility Models: General Policies for Zero-Shot Deployment in New Environmentscited by 43
  22. 2024Vision-Language Models Provide Promptable Representations for Reinforcement Learningcited by 41
  23. 2022History Compression via Language Models in Reinforcement Learningcited by 39
  24. 2023Learning adaptive planning representations with natural language guidancecited by 36
  25. 2021Reason first, then respond: Modular Generation for Knowledge-infused Dialoguecited by 32
  26. 2023Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agentscited by 32
  27. 2024Planetarium: A Rigorous Benchmark for Translating Text to Structured Planning Languagescited by 32
  28. 2024LLaMAR: Long-Horizon Planning for Multi-Agent Robots in Partially Observable Environmentscited by 32
  29. 2022Reconstructing Action-Conditioned Human-Object Interactions Using Commonsense Knowledge Priorscited by 30
  30. 2023Distilling Internet-Scale Vision-Language Models into Embodied Agentscited by 30
  31. 2023Thought Cloning: Learning to Think while Acting by Imitating Human Thinkingcited by 30
  32. 2023Vision-Language Interpreter for Robot Task Planningcited by 29
  33. 2023Interactive Planning Using Large Language Models for Partially Observable Robotics Taskscited by 27
  34. 2024DataEnvGym: Data Generation Agents in Teacher Environments with Student Feedbackcited by 23
  35. 2022How to talk so AI will learn: Instructions, descriptions, and autonomycited by 22
  36. 2022Generating Executable Action Plans with Environmentally-Aware Language Modelscited by 21
  37. 2023Human-like Few-Shot Learning via Bayesian Reasoning over Natural Languagecited by 21
  38. 2024Explaining Datasets in Words: Statistical Models with Natural Language Parameterscited by 21

50 most cited of 99 All 99 (JSON)