mlsanityData to 7 Oct 2026

TEACh: Task-driven Embodied Agents that Chat

arXiv 2110.00534 ↗First posted 1 Oct 2021cs.CV cs.AI cs.CLcited by 194

Not on the map. Why?

Cited by 32 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Robots operating in human spaces must be able to engage in natural language interaction with people, both understanding and executing instructions, and using conversation to resolve ambiguity and recover from mistakes. To study this, we introduce TEACh, a dataset of over 3,000 human--human, interactive dialogues to complete household tasks in simulation. A Commander with access to oracle information about a task communicates in natural language with a Follower. The Follower navigates through and interacts with the environment to complete tasks varying in complexity from "Make Coffee" to "Prepare Breakfast", asking questions and getting additional information from the Commander. We propose three benchmarks using TEACh to study embodied intelligence challenges, and we evaluate initial models' abilities in dialogue understanding, language grounding, and task execution.

Key facts. Cited by 194 papers in the map (data through 7 Oct 2026). Builds on 26 of its 49 references that are in the map. First posted on arXiv 1 Oct 2021.

Builds on

26 of 49 references.

References · 26

  1. 2017Mask R-CNNcited by 7,259
  2. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  3. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  4. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  5. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  6. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  7. 2019Vision-and-Dialog Navigationcited by 246
  8. 2020Experience Grounds Languagecited by 241
  9. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
  10. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  11. 2018Mapping Instructions to Actions in 3D Environments with Visual Goal Predictioncited by 101
  12. 2019Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learningcited by 100
14 more
  1. 2021A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Executioncited by 91
  2. 2019Executing Instructions in Situated Collaborative Interactionscited by 68
  3. 2021Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completioncited by 57
  4. 2019Just Ask:An Interactive Learning Framework for Vision and Language Navigationcited by 53
  5. 2021Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoringcited by 53
  6. 2015Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequencescited by 50
  7. 2020BabyWalk: Going Farther in Vision-and-Language Navigation by Taking Baby Stepscited by 49
  8. 2018Mapping Navigation Instructions to Continuous Control Actions with Position-Visitation Predictioncited by 35
  9. 2020RMM: A Recursive Mental Model for Dialog Navigationcited by 32
  10. 2018Shifting the Baseline: Single Modality Performance on Visual Navigation & QAcited by 30
  11. 2021The MineRL BASALT Competition on Learning from Human Feedbackcited by 24
  12. 2017Learning Interpretable Spatial Operations in a Rich 3D Blocks Worldcited by 15
  13. 2020ArraMon: A Joint Navigation-Assembly Instruction Interpretation Task in Dynamic Environmentscited by 14
  14. 2021VISITRON: Visual Semantics-Aligned Interactively Trained Object-Navigatorcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 26; 2023: 45; 2024: 41; 2025: 42; 2026: 38.

194 papers.

Sort:

By area:

  1. 2023Cognitive Architectures for Language Agentscited by 422
  2. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  3. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 350
  4. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
  5. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 231
  6. 2023Habitat 3.0: A Co-Habitat for Humans, Avatars and Robotscited by 220
  7. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 201
  8. 2023HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real Worldcited by 159
  9. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 116
  10. 2022Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scalecited by 116
  11. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 115
  12. 2022EgoTaskQA: Understanding Human Tasks in Egocentric Videoscited by 112
38 more
  1. 2023Discuss Before Moving: Visual Language Navigation via Multi-expert Discussionscited by 108
  2. 2023HomeRobot: Open-Vocabulary Mobile Manipulationcited by 104
  3. 2024EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Modelscited by 100
  4. 2023MindAgent: Emergent Gaming Interactioncited by 87
  5. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  6. 2023PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigationcited by 77
  7. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  8. 2023ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenescited by 68
  9. 2023Learning to Model the World with Languagecited by 66
  10. 2022DialFRED: Dialogue-Enabled Agents for Embodied Instruction Followingcited by 63
  11. 2022Housekeep: Tidying Virtual Households using Commonsense Reasoningcited by 57
  12. 2022CLIP-Nav: Using CLIP for Zero-Shot Vision-and-Language Navigationcited by 55
  13. 2025Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigationcited by 50
  14. 2022Learning from Unlabeled 3D Environments for Vision-and-Language Navigationcited by 49
  15. 2022Aerial Vision-and-Dialog Navigationcited by 48
  16. 2024COMBO: Compositional World Models for Embodied Multi-Agent Cooperationcited by 47
  17. 2022Retrospectives on the Embodied AI Workshopcited by 46
  18. 2023EmotionGesture: Audio-Driven Diverse Emotional Co-Speech 3D Gesture Generationcited by 40
  19. 2023Learning Navigational Visual Representations with Semantic Map Supervisioncited by 40
  20. 2025Vision Language Action Models in Robotic Manipulation: A Systematic Reviewcited by 38
  21. 2023Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Modelscited by 37
  22. 2023Context-Aware Planning and Environment-Aware Memory for Instruction Following Embodied Agentscited by 33
  23. 2024Asking the Right Question at the Right Time: Human and Model Uncertainty Guidance to Ask Clarification Questionscited by 32
  24. 2024LLaMAR: Long-Horizon Planning for Multi-Agent Robots in Partially Observable Environmentscited by 32
  25. 2023Improving Grounded Language Understanding in a Collaborative Environment by Interacting with Agents Through Help Feedbackcited by 31
  26. 2022TIDEE: Tidying Up Novel Rooms using Visuo-Semantic Commonsense Priorscited by 30
  27. 2022JARVIS: A Neuro-Symbolic Commonsense Reasoning Framework for Conversational Embodied Agentscited by 29
  28. 2022Neuro-Symbolic Procedural Planning with Commonsense Promptingcited by 28
  29. 2024DriVLMe: Enhancing LLM-based Autonomous Driving Agents with Embodied and Social Experiencescited by 27
  30. 2022Last-Mile Embodied Visual Navigationcited by 25
  31. 2024ODIN: A Single Model for 2D and 3D Segmentationcited by 25
  32. 2025Embodied AI: From LLMs to World Modelscited by 23
  33. 2022FOAM: A Follower-aware Speaker Model For Vision-and-Language Navigationcited by 22
  34. 2022I Cast Detect Thoughts: Learning to Converse and Guide with Intents and Theory-of-Mind in Dungeons and Dragonscited by 22
  35. 2024Online Continual Learning For Interactive Instruction Following Agentscited by 21
  36. 2025Grounding Multimodal LLMs to Embodied Agents that Ask for Help with Reinforcement Learningcited by 21
  37. 2023Incremental Learning of Humanoid Robot Behavior from Natural Interaction and Large Language Modelscited by 19
  38. 2023HandMeThat: Human-Robot Communication in Physical and Social Environmentscited by 19

50 most cited of 194 All 194 (JSON)