mlsanityData to 7 Oct 2026

Talk the Walk: Navigating New York City through Grounded Dialogue

arXiv 1807.03367 ↗First posted 9 Jul 2018cs.AI cs.CL cs.CVcited by 44

Not on the map. Why?

Abstract (arXiv)

We introduce "Talk The Walk", the first large-scale dialogue dataset grounded in action and perception. The task involves two agents (a "guide" and a "tourist") that communicate via natural language in order to achieve a common goal: having the tourist navigate to a given target location. The task and dataset, which are described in detail, are challenging and their full solution is an open problem that we pose to the community. We (i) focus on the task of tourist localization and develop the novel Masked Attention for Spatial Convolutions (MASC) mechanism that allows for grounding tourist utterances into the guide's map, (ii) show it yields significant improvements for both emergent and natural language communication, and (iii) using this method, we establish non-trivial baselines on the full task.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Jul 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 1; 2020: 0; 2021: 12; 2022: 8; 2023: 7; 2024: 6; 2025: 4; 2026: 5.

44 papers.

Sort:

By area:

  1. 2023Towards Learning a Generalist Model for Embodied Navigationcited by 135
  2. 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 118
  3. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 116
  4. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 115
  5. 2019Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learningcited by 100
  6. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  7. 2018Vision-based Navigation with Language-based Assistance via Imitation Learning with Indirect Interventioncited by 77
  8. 2021Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completioncited by 57
  9. 2021SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversationscited by 54
  10. 2022Learning from Unlabeled 3D Environments for Vision-and-Language Navigationcited by 49
  11. 2021Vision-Language Navigation: A Survey and Taxonomycited by 33
  12. 2022Episodic Memory Question Answeringcited by 32
32 more
  1. 2021Core Challenges in Embodied Vision-Language Planningcited by 30
  2. 2022Multimodal Conversational AI: A Survey of Datasets and Approachescited by 16
  3. 2021Reference-Centric Models for Grounded Collaborative Dialoguecited by 15
  4. 2021A Framework for Learning to Request Rich and Contextually Useful Information from Humanscited by 12
  5. 2021Maintaining Common Ground in Dynamic Environmentscited by 9
  6. 2024Vision-Language Navigation with Embodied Intelligence: A Surveycited by 9
  7. 2022DOROTHIE: Spoken Dialogue for Handling Unexpected Situations in Interactive Autonomous Driving Agentscited by 8
  8. 2024Conversational Grounding: Annotation and Analysis of Grounding Acts and Grounding Unitscited by 8
  9. 2025CitySeeker: How Do VLMS Explore Embodied Urban Navigation With Implicit Human Needs?cited by 8
  10. 2022Over-communicate no more: Situated RL agents learn concise communication protocolscited by 4
  11. 2024SCOUT: A Situated and Multi-Modal Human-Robot Dialogue Corpuscited by 4
  12. 2023ELBA: Learning by Asking for Embodied Visual Navigation and Task Completioncited by 3
  13. 2023HeGeL: A Novel Dataset for Geo-Location from Hebrew Textcited by 3
  14. 2023Towards a Neural Era in Dialogue Management for Collaboration: A Literature Surveycited by 3
  15. 2024Where Do We Go from Here? Multi-scale Allocentric Relational Inference from Natural Spatial Descriptionscited by 3
  16. 2025DialNav: Multi-turn Dialog Navigation with a Remote Guidecited by 3
  17. 2021Deep Learning for Embodied Vision Navigation: A Surveycited by 2
  18. 2022ViLPAct: A Benchmark for Compositional Generalization on Multimodal Human Activitiescited by 2
  19. 2021gComm: An environment for investigating generalization in Grounded Language Acquisitioncited by 1
  20. 2024Into the Unknown: Generating Geospatial Descriptions for New Environmentscited by 1
  21. 2026A Comprehensive Survey and Systematic Real-World Evaluation of Embodied Vision-and-Language Navigationcited by 1
  22. 2021Collision Replay: What Does Bumping Into Things Tell You About Scene Geometry?cited by 0
  23. 2021Dialogue Object Searchcited by 0
  24. 2022Navigating Connected Memories with a Task-oriented Dialog Systemcited by 0
  25. 2023Core Challenges in Embodied Vision-Language Planningcited by 0
  26. 2023Navigating to Success in Multi-Modal Human-Robot Collaboration: Analysis and Corpus Releasecited by 0
  27. 2025Co-Alignment: Rethinking Alignment as Bidirectional Human-AI Cognitive Adaptationcited by 0
  28. 2025Conversational Orientation Reasoning: Egocentric-to-Allocentric Navigation with Multimodal Chain-of-Thoughtcited by 0
  29. 2026Guide Me Out: A Framework to Benchmark VLM Operators Communication in Crisis Scenarioscited by 0
  30. 2026Advancing DialNav through Automatic Embodied Dialog Augmentationcited by 0
  31. 2026NavVerse: Benchmarking Indoor-to-Outdoor Embodied Navigation in Continuous Robot Simulationcited by 0
  32. 2026Time-Aware Assistive Navigationcited by 0