mlsanityData to 7 Oct 2026

Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoring

arXiv 2106.03427 ↗First posted 7 Jun 2021cs.AI cs.CLcited by 53

Not on the map. Why?

Abstract (arXiv)

Despite recent progress, learning new tasks through language instructions remains an extremely challenging problem. On the ALFRED benchmark for task learning, the published state-of-the-art system only achieves a task success rate of less than 10% in an unseen environment, compared to the human performance of over 90%. To address this issue, this paper takes a closer look at task learning. In a departure from a widely applied end-to-end architecture, we decomposed task learning into three sub-problems: sub-goal planning, scene navigation, and object manipulation; and developed a model HiTUT (stands for Hierarchical Tasks via Unified Transformers) that addresses each sub-problem in a unified manner to learn a hierarchical task structure. On the ALFRED benchmark, HiTUT has achieved the best performance with a remarkably higher generalization ability. In the unseen environment, HiTUT achieves over 160% performance gain in success rate compared to the previous state of the art. The explicit representation of task structures also enables an in-depth understanding of the nature of the problem and the ability of the agent, which provides insight for future benchmark development and evaluation.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Builds on 30 of its 46 references that are in the map. First posted on arXiv 7 Jun 2021.

Builds on

30 of 46 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2017Mask R-CNNcited by 7,259
  6. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  7. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  8. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  9. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  10. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 349
  11. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  12. 2017Embodied Question Answeringcited by 335
18 more
  1. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  2. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  3. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
  4. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
  5. 2017IQA: Visual Question Answering in Interactive Environmentscited by 154
  6. 2019Stay on the Path: Instruction Fidelity in Vision-and-Language Navigationcited by 129
  7. 2019The Regretful Agent: Heuristic-Aided Navigation through Progress Estimationcited by 104
  8. 2018Mapping Instructions to Actions in 3D Environments with Visual Goal Predictioncited by 101
  9. 2019Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigationcited by 81
  10. 2017Mapping Instructions and Visual Observations to Actions with Reinforcement Learningcited by 80
  11. 2018Neural Modular Control for Embodied Question Answeringcited by 56
  12. 2017Visual Semantic Planning using Deep Successor Representationscited by 45
  13. 2019Learning to Map Natural Language Instructions to Physical Quadcopter Control using Simulated Flightcited by 43
  14. 2020Diagnosing the Environment Bias in Vision-and-Language Navigationcited by 41
  15. 2020Visually-Grounded Planning without Vision: Language Models Infer Detailed Plans from High-level Instructionscited by 36
  16. 2019Situational Fusion of Visual Representation for Visual Navigationcited by 28
  17. 2020Learning Adaptive Language Interfaces through Decompositioncited by 22
  18. 2021Are We There Yet? Learning to Localize in Embodied Instruction Followingcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 11; 2023: 19; 2024: 13; 2025: 1; 2026: 2.

53 papers.

Sort:

By area:

  1. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  2. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  3. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  4. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  5. 2021TEACh: Task-driven Embodied Agents that Chatcited by 194
  6. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  7. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 129
  8. 2021FILM: Following Instructions in Language with Modular Methodscited by 119
  9. 2021Skill Induction and Planning with Latent Languagecited by 99
  10. 2021A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Executioncited by 91
  11. 2023ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenescited by 68
  12. 2023DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignmentcited by 58
38 more
  1. 2021Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completioncited by 57
  2. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 43
  3. 2023Context-Aware Planning and Environment-Aware Memory for Instruction Following Embodied Agentscited by 33
  4. 2023Thought Cloning: Learning to Think while Acting by Imitating Human Thinkingcited by 30
  5. 2022One Step at a Time: Long-Horizon Vision-and-Language Navigation with Milestonescited by 26
  6. 2022Few-shot Subgoal Planning with Language Modelscited by 22
  7. 2023InstructABSA: Instruction Learning for Aspect Based Sentiment Analysiscited by 22
  8. 2023Instruction Tuned Models are Quick Learnerscited by 21
  9. 2023ThinkBot: Embodied Instruction Following with Thought Chain Reasoningcited by 19
  10. 2023Chain-of-Thought Predictive Controlcited by 18
  11. 2022DANLI: Deliberative Agent for Following Natural Language Instructionscited by 15
  12. 2024P-RAG: Progressive Retrieval Augmented Generation For Planning on Embodied Everyday Taskcited by 14
  13. 2023ANPL: Towards Natural Programming with Interactive Decompositioncited by 13
  14. 2021LUMINOUS: Indoor Scene Generation for Embodied AI Challengescited by 12
  15. 2022LEBP -- Language Expectation & Binding Policy: A Two-Stream Framework for Embodied Vision-and-Language Interaction Task Learning Agentscited by 11
  16. 2023Embodied Concept Learner: Self-supervised Learning of Concepts and Mapping through Instruction Followingcited by 9
  17. 2023MAEA: Multimodal Attribution for Embodied AIcited by 9
  18. 2024GROOT-2: Weakly Supervised Multi-Modal Instruction Following Agentscited by 9
  19. 2022Learning to Act with Affordance-Aware Multimodal Neural SLAMcited by 8
  20. 2022Moment-based Adversarial Training for Embodied Language Comprehensioncited by 8
  21. 2022DOROTHIE: Spoken Dialogue for Handling Unexpected Situations in Interactive Autonomous Driving Agentscited by 8
  22. 2023LACMA: Language-Aligning Contrastive Learning with Meta-Actions for Embodied Instruction Followingcited by 8
  23. 2024HELPER-X: A Unified Instructable Embodied Agent to Tackle Four Interactive Vision-Language Domains with Memory-Augmented Language Modelscited by 8
  24. 2024Embodied Instruction Following in Unknown Environmentscited by 8
  25. 2023Egocentric Planning for Scalable Embodied Task Achievementcited by 7
  26. 2023Multi-Level Compositional Reasoning for Interactive Instruction Followingcited by 7
  27. 2024Socratic Planner: Self-QA-Based Zero-Shot Planning for Embodied Instruction Followingcited by 7
  28. 2024Teaching Embodied Reinforcement Learning Agents: Informativeness and Diversity of Language Usecited by 7
  29. 2023Heterogeneous Embodied Multi-Agent Collaborationcited by 6
  30. 2022See, Plan, Predict: Language-guided Cognitive Planning with Video Predictioncited by 4
  31. 2024DISCO: Embodied Navigation and Interaction via Differentiable Scene Semantics and Dual-level Controlcited by 4
  32. 2023Less is more -- the Dispatcher/ Executor principle for multi-task Reinforcement Learningcited by 2
  33. 2024RoboCoder: Robotic Learning from Basic Skills to General Tasks with Large Language Modelscited by 2
  34. 2024Causality-Aware Transformer Networks for Robotic Navigationcited by 2
  35. 2023tagE: Enabling an Embodied Agent to Understand Human Instructionscited by 1
  36. 2024SEAL: SEmantic-Augmented Imitation Learning via Language Modelcited by 1
  37. 2021Are you doing what I say? On modalities alignment in ALFREDcited by 0
  38. 2024ET tu, CLIP? Addressing Common Object Errors for Unseen Environmentscited by 0

50 most cited of 53 All 53 (JSON)