Situated Instruction Following
Not on the map. Why?
Abstract (arXiv)
Language is never spoken in a vacuum. It is expressed, comprehended, and contextualized within the holistic backdrop of the speaker's history, actions, and environment. Since humans are used to communicating efficiently with situated language, the practicality of robotic assistants hinge on their ability to understand and act upon implicit and situated instructions. In traditional instruction following paradigms, the agent acts alone in an empty house, leading to language use that is both simplified and artificially "complete." In contrast, we propose situated instruction following, which embraces the inherent underspecification and ambiguity of real-world communication with the physical presence of a human speaker. The meaning of situated instructions naturally unfold through the past actions and the expected future behaviors of the human involved. Specifically, within our settings we have instructions that (1) are ambiguously specified, (2) have temporally evolving intent, (3) can be interpreted more precisely with the agent's dynamic actions. Our experiments indicate that state-of-the-art Embodied Instruction Following (EIF) models lack holistic understanding of situated human intention.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 62 of its 76 references that are in the map. First posted on arXiv 15 Jul 2024.
Builds on
62 of 76 references.
References · 62
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2023GPT-4 Technical Reportcited by 18,396
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
- 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
- 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
38 more
- 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
- 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
- 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
- 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
- 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
- 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 614
- 2022Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Itemscited by 612
- 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 611
- 2018On Evaluation of Embodied Navigation Agentscited by 547
- 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 462
- 2018Scalable agent alignment via reward modeling: a research directioncited by 447
- 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
- 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 429
- 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 414
- 2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 366
- 2021ABO: Dataset and Benchmarks for Real-World 3D Object Understandingcited by 366
- 2017Embodied Question Answeringcited by 336
- 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 333
- 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 282
- 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 280
- 2023Habitat 3.0: A Co-Habitat for Humans, Avatars and Robotscited by 228
- 2024STAR: A Benchmark for Situated Reasoning in Real-World Videoscited by 213
- 2021TEACh: Task-driven Embodied Agents that Chatcited by 197
- 2023ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigationcited by 127
- 2021FILM: Following Instructions in Language with Modular Methodscited by 121
- 2023L3MVN: Leveraging Large Language Models for Visual Target Navigationcited by 120
- 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 117
- 2023Evil Geniuses: Delving into the Safety of LLM-based Agentscited by 106
- 2021Visual Room Rearrangementcited by 102
- 2023Clarify When Necessary: Resolving Ambiguity Through Interaction with LMscited by 98
- 2023Eliciting Human Preferences with Language Modelscited by 92
- 2019Executing Instructions in Situated Collaborative Interactionscited by 69
- 2021Memory-Augmented Reinforcement Learning for Image-Goal Navigationcited by 66
- 2022DialFRED: Dialogue-Enabled Agents for Embodied Instruction Followingcited by 63
- 2022Instance-Specific Image Goal Navigation: Training Embodied Agents to Find Object Instancescited by 55
- 2023LLM-Powered Hierarchical Language Agent for Real-time Human-AI Coordinationcited by 48
- 2023Train Offline, Test Online: A Real Robot Learning Benchmarkcited by 47
50 most cited of 62 All 62 (JSON)
Built on by
Citing papers per year: 2025: 3.
3 papers.