mlsanityData to 7 Oct 2026

Speaker-Follower Models for Vision-and-Language Navigation

arXiv 1806.02724 ↗First posted 7 Jun 2018cs.CV cs.CLcited by 337

Not on the map. Why?

Abstract (arXiv)

Navigation guided by natural language instructions presents a challenging reasoning problem for instruction followers. Natural language instructions typically identify only a few high-level decisions and landmarks rather than complete low-level motor behaviors; much of the missing information must be inferred based on perceptual context. In machine learning settings, this is doubly challenging: it is difficult to collect enough annotated data to enable learning of this reasoning process from scratch, and also difficult to implement the reasoning process using generic sequence models. Here we describe an approach to vision-and-language navigation that addresses both these issues with an embedded speaker model. We use this speaker model to (1) synthesize new instructions for data augmentation and to (2) implement pragmatic reasoning, which evaluates how well candidate action sequences explain an instruction. Both steps are supported by a panoramic action space that reflects the granularity of human-generated instructions. Experiments show that all three components of this approach---speaker-driven data augmentation, pragmatic reasoning and panoramic action space---dramatically improve the performance of a baseline instruction follower, more than doubling the success rate over the best existing approach on a standard benchmark.

Key facts. Cited by 337 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Jun 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 2; 2019: 1; 2020: 1; 2021: 59; 2022: 59; 2023: 57; 2024: 66; 2025: 42; 2026: 50.

337 papers.

Sort:

By area:

  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  2. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  3. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  4. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  5. 2022Visual Language Maps for Robot Navigationcited by 362
  6. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  7. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  8. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  9. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  10. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  11. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  12. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
38 more
  1. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  2. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 201
  3. 2021TEACh: Task-driven Embodied Agents that Chatcited by 194
  4. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
  5. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 165
  6. 2021BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environmentscited by 161
  7. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  8. 2023Towards Learning a Generalist Model for Embodied Navigationcited by 135
  9. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 129
  10. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 129
  11. 2022Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigationcited by 127
  12. 2021Waypoint Models for Instruction-guided Navigation in Continuous Environmentscited by 125
  13. 2021FILM: Following Instructions in Language with Modular Methodscited by 119
  14. 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 118
  15. 2021Pathdreamer: A World Model for Indoor Navigationcited by 117
  16. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 117
  17. 2023GridMM: Grid Memory Map for Vision-and-Language Navigationcited by 117
  18. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 116
  19. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 115
  20. 2023Discuss Before Moving: Visual Language Navigation via Multi-expert Discussionscited by 108
  21. 2019Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learningcited by 100
  22. 2024NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Modelscited by 96
  23. 2021Structured Scene Memory for Vision-Language Navigationcited by 95
  24. 2021A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Executioncited by 91
  25. 2022Cross-modal Map Learning for Vision and Language Navigationcited by 91
  26. 2021LanguageRefer: Spatial-Language Model for 3D Visual Groundingcited by 88
  27. 2021Language-Aligned Waypoint (LAW) Supervision for Vision-and-Language Navigation in Continuous Environmentscited by 87
  28. 2022Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigationcited by 86
  29. 2024MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigationcited by 85
  30. 2022EnvEdit: Environment Editing for Vision-and-Language Navigationcited by 83
  31. 2022Coder Reviewer Reranking for Code Generationcited by 83
  32. 2023DREAMWALKER: Mental Planning for Continuous Vision-Language Navigationcited by 82
  33. 2022Sim-2-Sim Transfer for Vision-and-Language Navigation in Continuous Environmentscited by 80
  34. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  35. 2018Vision-based Navigation with Language-based Assistance via Imitation Learning with Indirect Interventioncited by 77
  36. 2023PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigationcited by 77
  37. 2024NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoningcited by 77
  38. 2023Bird's-Eye-View Scene Graph for Vision-Language Navigationcited by 71

50 most cited of 337 All 337 (JSON)