mlsanityData to 7 Oct 2026

Executing Instructions in Situated Collaborative Interactions

arXiv 1910.03655 ↗First posted 8 Oct 2019cs.CL cs.AI cs.LGcited by 68

Not on the map. Why?

Abstract (arXiv)

We study a collaborative scenario where a user not only instructs a system to complete tasks, but also acts alongside it. This allows the user to adapt to the system abilities by changing their language or deciding to simply accomplish some tasks themselves, and requires the system to effectively recover from errors as the user strategically assigns it new goals. We build a game environment to study this scenario, and learn to map user instructions to system actions. We introduce a learning approach focused on recovery from cascading errors between instructions, and modeling methods to explicitly reason about instructions with multiple goals. We evaluate with a new evaluation protocol using recorded interactions and online games with human users, and observe how users adapt to the system abilities.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 17; 2022: 15; 2023: 15; 2024: 8; 2025: 6; 2026: 6.

68 papers.

Sort:

By area:

  1. 2021TEACh: Task-driven Embodied Agents that Chatcited by 194
  2. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 116
  3. 2023MindAgent: Emergent Gaming Interactioncited by 87
  4. 2021Action-Based Conversations Dataset: A Corpus for Building More In-Depth Task-Oriented Dialogue Systemscited by 73
  5. 2021MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Taskscited by 62
  6. 2021Less is More: Generating Grounded Navigation Instructions from Landmarkscited by 62
  7. 2020Learning Rewards from Linguistic Feedbackcited by 41
  8. 2021Communicating Natural Programs to Humans and Machinescited by 35
  9. 2021Vision-Language Navigation: A Survey and Taxonomycited by 33
  10. 2021Core Challenges in Embodied Vision-Language Planningcited by 30
  11. 2022Learning to Execute Actions or Ask Clarification Questionscited by 30
  12. 2023Decision-Oriented Dialogue for Human-AI Collaborationcited by 29
38 more
  1. 2021Look Wide and Interpret Twice: Improving Performance on Interactive Instruction-following Taskscited by 25
  2. 2022Last-Mile Embodied Visual Navigationcited by 25
  3. 2022Few-shot Subgoal Planning with Language Modelscited by 22
  4. 2022Continual Learning for Instruction Following from Realtime Feedbackcited by 21
  5. 2021Continual Learning for Grounded Instruction Generation by Observing Human Following Behaviorcited by 19
  6. 2023HandMeThat: Human-Robot Communication in Physical and Social Environmentscited by 19
  7. 2023Alexa Arena: A User-Centric Interactive Platform for Embodied AIcited by 18
  8. 2023Meta-Explore: Exploratory Hierarchical Vision-and-Language Navigation Using Scene Object Spectrum Groundingcited by 18
  9. 2021Learning to communicate about shared procedural abstractionscited by 15
  10. 2021Reference-Centric Models for Grounded Collaborative Dialoguecited by 15
  11. 2021Draw Me a Flower: Processing and Grounding Abstraction in Natural Languagecited by 14
  12. 2024Efficacy of Language Model Self-Play in Non-Zero-Sum Gamescited by 14
  13. 2022Workflow Discovery from Dialogues in the Low Data Regimecited by 12
  14. 2022Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approachescited by 12
  15. 2022Don't Copy the Teacher: Data and Model Challenges in Embodied Dialoguecited by 10
  16. 2021Maintaining Common Ground in Dynamic Environmentscited by 9
  17. 2021Analysis of Language Change in Collaborative Instruction Followingcited by 9
  18. 2022MUG: Interactive Multimodal Grounding on User Interfacescited by 9
  19. 2024Vision-Language Navigation with Embodied Intelligence: A Surveycited by 9
  20. 2024Infer Human's Intentions Before Following Natural Language Instructionscited by 9
  21. 2024Autonomous Workflow for Multimodal Fine-Grained Training Assistants Towards Mixed Realitycited by 8
  22. 2025AmbiK: Dataset of Ambiguous Tasks in Kitchen Environmentcited by 8
  23. 2022LEMON: Language-Based Environment Manipulation via Execution-Guided Pre-trainingcited by 5
  24. 2022Evaluating Multimodal Interactive Agentscited by 5
  25. 2023"We Demand Justice!": Towards Social Context Grounding of Political Textscited by 4
  26. 2024SCOUT: A Situated and Multi-Modal Human-Robot Dialogue Corpuscited by 4
  27. 2021PanGEA: The Panoramic Graph Environment Annotation Toolkitcited by 3
  28. 2023ELBA: Learning by Asking for Embodied Visual Navigation and Task Completioncited by 3
  29. 2023Multimodal Speech Recognition for Language-Guided Embodied Agentscited by 3
  30. 2024Situated Instruction Followingcited by 3
  31. 2025DialNav: Multi-turn Dialog Navigation with a Remote Guidecited by 3
  32. 2025Characterizing Language Use in a Collaborative Situated Gamecited by 3
  33. 2022ViLPAct: A Benchmark for Compositional Generalization on Multimodal Human Activitiescited by 2
  34. 2023Yes, this Way! Learning to Ground Referring Expressions into Actions with Intra-episodic Feedback from Supportive Teacherscited by 2
  35. 2025CrafText Benchmark: Advancing Instruction Following in Complex Multimodal Open-Ended Worldcited by 2
  36. 2025MDC-R: The Minecraft Dialogue Corpus with Referencecited by 2
  37. 2022Many Episode Learning in a Modular Embodied Agent via End-to-End Interactioncited by 1
  38. 2022A Computational Interface to Translate Strategic Intent from Unstructured Language in a Low-Data Settingcited by 1

50 most cited of 68 All 68 (JSON)