mlsanityData to 14 Oct 2026

ELBA: Learning by Asking for Embodied Visual Navigation and Task Completion

arXiv 2302.04865 ↗First posted 9 Feb 2023cs.CV cs.AI cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

The research community has shown increasing interest in designing intelligent embodied agents that can assist humans in accomplishing tasks. Although there have been significant advancements in related vision-language benchmarks, most prior work has focused on building agents that follow instructions rather than endowing agents the ability to ask questions to actively resolve ambiguities arising naturally in embodied environments. To address this gap, we propose an Embodied Learning-By-Asking (ELBA) model that learns when and what questions to ask to dynamically acquire additional information for completing the task. We evaluate ELBA on the TEACh vision-dialog navigation and task completion dataset. Experimental results show that the proposed method achieves improved task performance compared to baseline models without question-answering capabilities.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 36 of its 53 references that are in the map. First posted on arXiv 9 Feb 2023.

Builds on

36 of 53 references.

References · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2015VQA: Visual Question Answeringcited by 2,506
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  8. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  9. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
  10. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  11. 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 610
  12. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 340
24 more
  1. 2017Embodied Question Answeringcited by 336
  2. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 271
  3. 2019Vision-and-Dialog Navigationcited by 250
  4. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 231
  5. 2021TEACh: Task-driven Embodied Agents that Chatcited by 197
  6. 2021Episodic Transformer for Vision-and-Language Navigationcited by 156
  7. 2017IQA: Visual Question Answering in Interactive Environmentscited by 155
  8. 2020Rearrangement: A Challenge for Embodied AIcited by 155
  9. 2022MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuningcited by 109
  10. 2019Embodied Question Answering in Photorealistic Environments with Point Cloud Perceptioncited by 103
  11. 2021Visual Room Rearrangementcited by 102
  12. 2019Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learningcited by 100
  13. 2019Executing Instructions in Situated Collaborative Interactionscited by 69
  14. 2022DialFRED: Dialogue-Enabled Agents for Embodied Instruction Followingcited by 63
  15. 2016Generating Natural Questions About an Imagecited by 56
  16. 2019Just Ask:An Interactive Learning Framework for Vision and Language Navigationcited by 53
  17. 2018Talk the Walk: Navigating New York City through Grounded Dialoguecited by 44
  18. 2020Environment-agnostic Multitask Learning for Natural Language Grounded Navigationcited by 43
  19. 2020Diagnosing the Environment Bias in Vision-and-Language Navigationcited by 41
  20. 2020RMM: A Recursive Mental Model for Dialog Navigationcited by 32
  21. 2021Core Challenges in Embodied Vision-Language Planningcited by 30
  22. 2020Vision-Dialog Navigation by Exploring Cross-modal Memorycited by 23
  23. 2022Ask4Help: Learning to Leverage an Expert for Embodied Taskscited by 18
  24. 2021VISITRON: Visual Semantics-Aligned Interactively Trained Object-Navigatorcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 0; 2026: 1.

3 papers.

  1. 2024Asking the Right Question at the Right Time: Human and Model Uncertainty Guidance to Ask Clarification Questionscited by 32
  2. 2024Collaborative Instance Object Navigation: Leveraging Uncertainty-Awareness to Minimize Human-Agent Dialoguescited by 8
  3. 2026PRISM: Perception Reasoning Interleaved for Sequential Decision Makingcited by 0