mlsanityData to 7 Oct 2026

DialFRED: Dialogue-Enabled Agents for Embodied Instruction Following

arXiv 2202.13330 ↗First posted 27 Feb 2022cs.AI cs.ROcited by 63

Not on the map. Why?

Abstract (arXiv)

Language-guided Embodied AI benchmarks requiring an agent to navigate an environment and manipulate objects typically allow one-way communication: the human user gives a natural language command to the agent, and the agent can only follow the command passively. We present DialFRED, a dialogue-enabled embodied instruction following benchmark based on the ALFRED benchmark. DialFRED allows an agent to actively ask questions to the human user; the additional information in the user's response is used by the agent to better complete its task. We release a human-annotated dataset with 53K task-relevant questions and answers and an oracle to answer questions. To solve DialFRED, we propose a questioner-performer framework wherein the questioner is pre-trained with the human-annotated data and fine-tuned with reinforcement learning. We make DialFRED publicly available and encourage researchers to propose and evaluate their solutions to building dialog-enabled embodied agents.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Builds on 28 of its 42 references that are in the map. First posted on arXiv 27 Feb 2022.

Builds on

28 of 42 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  3. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  4. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  5. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  6. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  7. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  8. 2017Embodied Question Answeringcited by 335
  9. 2020A Simple Language Model for Task-Oriented Dialoguecited by 331
  10. 2020ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulationcited by 249
  11. 2019Vision-and-Dialog Navigationcited by 246
  12. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 240
16 more
  1. 2021TEACh: Task-driven Embodied Agents that Chatcited by 194
  2. 2018Neural Approaches to Conversational AIcited by 164
  3. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  4. 2017IQA: Visual Question Answering in Interactive Environmentscited by 154
  5. 2016Neural Belief Tracker: Data-Driven Dialogue State Trackingcited by 142
  6. 2021FILM: Following Instructions in Language with Modular Methodscited by 119
  7. 2019Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learningcited by 100
  8. 2021A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Executioncited by 91
  9. 2019Interactive Gibson Benchmark (iGibson 0.5): A Benchmark for Interactive Navigation in Cluttered Environmentscited by 89
  10. 2018Vision-based Navigation with Language-based Assistance via Imitation Learning with Indirect Interventioncited by 77
  11. 2019Just Ask:An Interactive Learning Framework for Vision and Language Navigationcited by 53
  12. 2017Composite Task-Completion Dialogue Policy Learning via Hierarchical Deep Reinforcement Learningcited by 44
  13. 2020RMM: A Recursive Mental Model for Dialog Navigationcited by 32
  14. 2019VRKitchen: an Interactive 3D Virtual Environment for Task-oriented Learningcited by 29
  15. 2016On-line Active Reward Learning for Policy Optimisation in Spoken Dialogue Systemscited by 28
  16. 2020Interactive Question Clarification in Dialogue via Reinforcement Learningcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 16; 2024: 20; 2025: 14; 2026: 7.

63 papers.

Sort:

By area:

  1. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
  2. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  3. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 116
  4. 2023HomeRobot: Open-Vocabulary Mobile Manipulationcited by 104
  5. 2023MindAgent: Emergent Gaming Interactioncited by 87
  6. 2023Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigationcited by 74
  7. 2023ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenescited by 68
  8. 20233D Concept Learning and Reasoning from Multi-View Imagescited by 60
  9. 2022CLIP-Nav: Using CLIP for Zero-Shot Vision-and-Language Navigationcited by 55
  10. 2022Aerial Vision-and-Dialog Navigationcited by 48
  11. 2023SPOC: Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real Worldcited by 47
  12. 2024Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generationcited by 44
38 more
  1. 2025Vision Language Action Models in Robotic Manipulation: A Systematic Reviewcited by 38
  2. 2025Multimodal Fusion and Vision-Language Models: A Survey for Robot Visioncited by 37
  3. 2024Online Continual Learning For Interactive Instruction Following Agentscited by 21
  4. 2025Grounding Multimodal LLMs to Embodied Agents that Ask for Help with Reinforcement Learningcited by 21
  5. 2023HandMeThat: Human-Robot Communication in Physical and Social Environmentscited by 19
  6. 2023Alexa Arena: A User-Centric Interactive Platform for Embodied AIcited by 18
  7. 2024Human-Aware Vision-and-Language Navigation: Bridging Simulation to Reality with Dynamic Human Interactionscited by 18
  8. 2026Theory of Space: Can Foundation Models Construct Spatial Beliefs through Active Exploration?cited by 18
  9. 2024PIVOT-R: Primitive-Driven Waypoint-Aware World Model for Robotic Manipulationcited by 17
  10. 2025VLNVerse: A Benchmark for Vision-Language Navigation with Versatile, Embodied, Realistic Simulation and Evaluationcited by 17
  11. 2024ReALFRED: An Embodied Instruction Following Benchmark in Photo-Realistic Environmentscited by 16
  12. 2025A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AIcited by 16
  13. 2023Instruction Clarification Requests in Multimodal Collaborative Dialogue Games: Tasks, and an Analysis of the CoDraw Datasetcited by 11
  14. 2025Offline RL by Reward-Weighted Fine-Tuning for Conversation Optimizationcited by 10
  15. 2024Vision-Language Navigation with Embodied Intelligence: A Surveycited by 9
  16. 2024"Don't forget to put the milk back!" Dataset for Enabling Embodied Agents to Detect Anomalous Situationscited by 9
  17. 2024Infer Human's Intentions Before Following Natural Language Instructionscited by 9
  18. 2022DOROTHIE: Spoken Dialogue for Handling Unexpected Situations in Interactive Autonomous Driving Agentscited by 8
  19. 2024HELPER-X: A Unified Instructable Embodied Agent to Tackle Four Interactive Vision-Language Domains with Memory-Augmented Language Modelscited by 8
  20. 2023LEMMA: Learning Language-Conditioned Multi-Robot Manipulationcited by 7
  21. 2024Collaborative Instance Object Navigation: Leveraging Uncertainty-Awareness to Minimize Human-Agent Dialoguescited by 7
  22. 2025REI-Bench: Can Embodied Agents Understand Vague Human Instructions in Task Planning?cited by 7
  23. 2024Simulating User Agents for Embodied Conversational-AIcited by 6
  24. 2025HA-VLN 2.0: An Open Benchmark and Leaderboard for Human-Aware Navigation in Discrete and Continuous Environments with Dynamic Multi-Human Interactionscited by 6
  25. 2023R2H: Building Multimodal Navigation Helpers that Respond to Help Requestscited by 5
  26. 2024Efficient Policy Adaptation with Contrastive Prompt Ensemble for Embodied Agentscited by 5
  27. 2024DISCO: Embodied Navigation and Interaction via Differentiable Scene Semantics and Dual-level Controlcited by 4
  28. 2025Ask-to-Clarify: Resolving Instruction Ambiguity through Multi-turn Dialoguecited by 4
  29. 2022Good Time to Ask: A Learning Framework for Asking for Help in Embodied Visual Navigationcited by 3
  30. 2022OpenD: A Benchmark for Language-Driven Door and Drawer Openingcited by 3
  31. 2023ELBA: Learning by Asking for Embodied Visual Navigation and Task Completioncited by 3
  32. 2024Situated Instruction Followingcited by 3
  33. 2025DialNav: Multi-turn Dialog Navigation with a Remote Guidecited by 3
  34. 2023Yes, this Way! Learning to Ground Referring Expressions into Actions with Intra-episodic Feedback from Supportive Teacherscited by 2
  35. 2024Hierarchical Spatial Proximity Reasoning for Vision-and-Language Navigationcited by 2
  36. 2025Learning to Generate Pointing Gestures in Situated Embodied Conversational Agentscited by 2
  37. 2025ESearch-R1: Learning Cost-Aware MLLM Agents for Interactive Embodied Search via Reinforcement Learningcited by 2
  38. 2026Ask When It Pays: Cost-Aware Open-Ended Interaction for Instance Goal Navigationcited by 2

50 most cited of 63 All 63 (JSON)