# ELBA: Learning by Asking for Embodied Visual Navigation and Task Completion

arXiv: https://arxiv.org/abs/2302.04865 · HTML: https://mlsanity.com/paper/2302.04865/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 53 references that are in the map.
- First posted on arXiv 9 Feb 2023.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds](https://mlsanity.com/paper/1906.03671/) (2019)
- [Speaker-Follower Models for Vision-and-Language Navigation](https://mlsanity.com/paper/1806.02724/) (2018)
- [Embodied Question Answering](https://mlsanity.com/paper/1711.11543/) (2017)
- [History Aware Multimodal Transformer for Vision-and-Language Navigation](https://mlsanity.com/paper/2110.13309/) (2021)
- [Vision-and-Dialog Navigation](https://mlsanity.com/paper/1907.04957/) (2019)
- [Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-training](https://mlsanity.com/paper/2002.10638/) (2020)
- [TEACh: Task-driven Embodied Agents that Chat](https://mlsanity.com/paper/2110.00534/) (2021)
- [Episodic Transformer for Vision-and-Language Navigation](https://mlsanity.com/paper/2105.06453/) (2021)
- [IQA: Visual Question Answering in Interactive Environments](https://mlsanity.com/paper/1712.03316/) (2017)
- [Rearrangement: A Challenge for Embodied AI](https://mlsanity.com/paper/2011.01975/) (2020)
- [MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning](https://mlsanity.com/paper/2212.10773/) (2022)
- [Embodied Question Answering in Photorealistic Environments with Point Cloud Perception](https://mlsanity.com/paper/1904.03461/) (2019)
- [Visual Room Rearrangement](https://mlsanity.com/paper/2103.16544/) (2021)
- [Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning](https://mlsanity.com/paper/1909.01871/) (2019)
- [Executing Instructions in Situated Collaborative Interactions](https://mlsanity.com/paper/1910.03655/) (2019)
- [DialFRED: Dialogue-Enabled Agents for Embodied Instruction Following](https://mlsanity.com/paper/2202.13330/) (2022)
- [Generating Natural Questions About an Image](https://mlsanity.com/paper/1603.06059/) (2016)
- [Just Ask:An Interactive Learning Framework for Vision and Language Navigation](https://mlsanity.com/paper/1912.00915/) (2019)
- [Talk the Walk: Navigating New York City through Grounded Dialogue](https://mlsanity.com/paper/1807.03367/) (2018)
- [Environment-agnostic Multitask Learning for Natural Language Grounded Navigation](https://mlsanity.com/paper/2003.00443/) (2020)

## Built on by (most cited first, 3 of 3)

- [Asking the Right Question at the Right Time: Human and Model Uncertainty Guidance to Ask Clarification Questions](https://mlsanity.com/paper/2402.06509/) (2024)
- [Collaborative Instance Object Navigation: Leveraging Uncertainty-Awareness to Minimize Human-Agent Dialogues](https://mlsanity.com/paper/2412.01250/) (2024)
- [PRISM: Perception Reasoning Interleaved for Sequential Decision Making](https://mlsanity.com/paper/2605.05407/) (2026)

## Abstract (arXiv)

The research community has shown increasing interest in designing intelligent embodied agents that can assist humans in accomplishing tasks. Although there have been significant advancements in related vision-language benchmarks, most prior work has focused on building agents that follow instructions rather than endowing agents the ability to ask questions to actively resolve ambiguities arising naturally in embodied environments. To address this gap, we propose an Embodied Learning-By-Asking (ELBA) model that learns when and what questions to ask to dynamically acquire additional information for completing the task. We evaluate ELBA on the TEACh vision-dialog navigation and task completion dataset. Experimental results show that the proposed method achieves improved task performance compared to baseline models without question-answering capabilities.