# Where Are You? Localization from Embodied Dialog

arXiv: https://arxiv.org/abs/2011.08277 · HTML: https://mlsanity.com/paper/2011.08277/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Nov 2020.

## Built on by (most cited first, 24 of 24)

- [Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey](https://mlsanity.com/paper/2105.04387/) (2021)
- [Pathdreamer: A World Model for Indoor Navigation](https://mlsanity.com/paper/2105.08756/) (2021)
- [Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scale](https://mlsanity.com/paper/2204.03514/) (2022)
- [A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learning](https://mlsanity.com/paper/2210.03112/) (2022)
- [TopViewRS: Vision-Language Models as Top-View Spatial Reasoners](https://mlsanity.com/paper/2406.02537/) (2024)
- [Episodic Memory Question Answering](https://mlsanity.com/paper/2205.01652/) (2022)
- [Last-Mile Embodied Visual Navigation](https://mlsanity.com/paper/2211.11746/) (2022)
- [GridToPix: Training Embodied Agents with Minimal Supervision](https://mlsanity.com/paper/2105.00931/) (2021)
- [RREx-BoT: Remote Referring Expressions with a Bag of Tricks](https://mlsanity.com/paper/2301.12614/) (2023)
- [Transformer-based Localization from Embodied Dialog with Large-scale Pre-training](https://mlsanity.com/paper/2210.04864/) (2022)
- [R2H: Building Multimodal Navigation Helpers that Respond to Help Requests](https://mlsanity.com/paper/2305.14260/) (2023)
- [DiaLoc: An Iterative Approach to Embodied Dialog Localization](https://mlsanity.com/paper/2403.06846/) (2024)
- [VPN: Visual Prompt Navigation](https://mlsanity.com/paper/2508.01766/) (2025)
- [PanGEA: The Panoramic Graph Environment Annotation Toolkit](https://mlsanity.com/paper/2103.12703/) (2021)
- [Which way is `right'?: Uncovering limitations of Vision-and-Language Navigation model](https://mlsanity.com/paper/2312.00151/) (2023)
- [DialNav: Multi-turn Dialog Navigation with a Remote Guide](https://mlsanity.com/paper/2509.12894/) (2025)
- [Deep Learning for Embodied Vision Navigation: A Survey](https://mlsanity.com/paper/2108.04097/) (2021)
- [The Wallpaper is Ugly: Indoor Localization using Vision and Language](https://mlsanity.com/paper/2410.03900/) (2024)
- [Neural Abstructions: Abstractions that Support Construction for Grounded Language Learning](https://mlsanity.com/paper/2107.09285/) (2021)
- [Top2Pano: Learning to Generate Indoor Panoramas from Top-Down View](https://mlsanity.com/paper/2507.21371/) (2025)
- [Global Commander and Local Operative: A Dual-Agent Framework for Scene Navigation](https://mlsanity.com/paper/2602.18941/) (2026)
- [Interactive Episodic Memory with User Feedback](https://mlsanity.com/paper/2604.24893/) (2026)
- [Advancing DialNav through Automatic Embodied Dialog Augmentation](https://mlsanity.com/paper/2606.19948/) (2026)
- [Time-Aware Assistive Navigation](https://mlsanity.com/paper/2609.05596/) (2026)

## Abstract (arXiv)

We present Where Are You? (WAY), a dataset of ~6k dialogs in which two humans -- an Observer and a Locator -- complete a cooperative localization task. The Observer is spawned at random in a 3D environment and can navigate from first-person views while answering questions from the Locator. The Locator must localize the Observer in a detailed top-down map by asking questions and giving instructions. Based on this dataset, we define three challenging tasks: Localization from Embodied Dialog or LED (localizing the Observer from dialog history), Embodied Visual Dialog (modeling the Observer), and Cooperative Localization (modeling both agents). In this paper, we focus on the LED task -- providing a strong baseline model with detailed ablations characterizing both dataset biases and the importance of various modeling choices. Our best model achieves 32.7% success at identifying the Observer's location within 3m in unseen buildings, vs. 70.4% for human Locators.