mlsanityData to 7 Oct 2026

Less is More: Generating Grounded Navigation Instructions from Landmarks

arXiv 2111.12872 ↗First posted 25 Nov 2021cs.CV cs.CLcited by 62

Not on the map. Why?

Abstract (arXiv)

We study the automatic generation of navigation instructions from 360-degree images captured on indoor routes. Existing generators suffer from poor visual grounding, causing them to rely on language priors and hallucinate objects. Our MARKY-MT5 system addresses this by focusing on visual landmarks; it comprises a first stage landmark detector and a second stage generator -- a multimodal, multilingual, multitask encoder-decoder. To train it, we bootstrap grounded landmark annotations on top of the Room-across-Room (RxR) dataset. Using text parsers, weak supervision from RxR's pose traces, and a multilingual image-text encoder trained on 1.8b images, we identify 971k English, Hindi and Telugu landmark descriptions and ground them to specific regions in panoramas. On Room-to-Room, human wayfinders obtain success rates (SR) of 71% following MARKY-MT5's instructions, just shy of their 75% SR following human instructions -- and well above SRs with other generators. Evaluations on RxR's longer, diverse paths obtain 61-64% SRs on three languages. Generating such high-quality navigation instructions in novel environments is a step towards conversational navigation tools and could facilitate larger-scale training of instruction-following agents.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Builds on 40 of its 68 references that are in the map. First posted on arXiv 25 Nov 2021.

Builds on

40 of 68 references.

References · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  8. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  9. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  10. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  11. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  12. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
28 more
  1. 2019Objects as Pointscited by 1,172
  2. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  3. 2016Stacked Hourglass Networks for Human Pose Estimationcited by 913
  4. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  5. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  6. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  7. 2018CornerNet: Detecting Objects as Paired Keypointscited by 763
  8. 2019Contrastive Representation Distillationcited by 734
  9. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  10. 2018On Evaluation of Embodied Navigation Agentscited by 526
  11. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  12. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  13. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  14. 2019Causal Confusion in Imitation Learningcited by 286
  15. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  16. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  17. 2019Connecting Vision and Language with Localized Narrativescited by 225
  18. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  19. 2020Language and Visual Entity Relationship Graph for Agent Navigationcited by 100
  20. 2019Executing Instructions in Situated Collaborative Interactionscited by 68
  21. 2019Universal Dependency Parsing from Scratchcited by 46
  22. 2021On the Evaluation of Vision-and-Language Navigation Instructionscited by 45
  23. 2020More Grounded Image Captioning by Distilling Image-Text Matching Modelcited by 43
  24. 2020Generative Language-Grounded Policy in Vision-and-Language Navigation with Bayes' Rulecited by 22
  25. 2021Continual Learning for Grounded Instruction Generation by Observing Human Following Behaviorcited by 19
  26. 2016Navigational Instruction Generation as Inverse Reinforcement Learning with Neural Machine Translationcited by 12
  27. 2021Multimodal Conditionality for Natural Language Generationcited by 6
  28. 2021PanGEA: The Panoramic Graph Environment Annotation Toolkitcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 16; 2024: 22; 2025: 11; 2026: 8.

62 papers.

Sort:

By area:

  1. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  2. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  3. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 165
  4. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 117
  5. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 115
  6. 2023Discuss Before Moving: Visual Language Navigation via Multi-expert Discussionscited by 108
  7. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  8. 2023PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigationcited by 77
  9. 2022A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learningcited by 60
  10. 2024Vision-and-Language Navigation via Causal Learningcited by 51
  11. 2023Learning Vision-and-Language Navigation from YouTube Videoscited by 47
  12. 2023Lana: A Language-Capable Navigator for Instruction Following and Generationcited by 39
38 more
  1. 2024DELTA: Decomposed Efficient Long-Term Robot Task Planning using Large Language Modelscited by 33
  2. 20221st Place Solutions for RxR-Habitat Vision-and-Language Navigation Competition (CVPR 2022)cited by 30
  3. 2023VLN-Trans: Translator for the Vision and Language Navigation Agentcited by 25
  4. 2023Fast-Slow Test-Time Adaptation for Online Vision-and-Language Navigationcited by 25
  5. 2024RoomTour3D: Geometry-Aware Video-Instruction Tuning for Embodied Navigationcited by 23
  6. 2024Controllable Navigation Instruction Generation with Chain of Thought Promptingcited by 20
  7. 2024Navigation Instruction Generation with BEV Perception and Large Language Modelscited by 20
  8. 2024Vision-Language Navigation with Energy-Based Policycited by 19
  9. 2024Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheelcited by 18
  10. 2025NavRAG: Generating User Demand Instructions for Embodied Navigation through Retrieval-Augmented LLMcited by 18
  11. 2025FlexVLN: Flexible Adaptation for Diverse Vision-and-Language Navigation Taskscited by 16
  12. 2025AeroDuo: Aerial Duo for UAV-based Vision and Language Navigationcited by 13
  13. 2024NavHint: Vision and Language Navigation Agent with a Hint Generatorcited by 12
  14. 2023Kefa: A Knowledge Enhanced and Fine-grained Aligned Speaker for Navigation Instruction Generationcited by 11
  15. 2024Navigating Beyond Instructions: Vision-and-Language Navigation in Obstructed Environmentscited by 10
  16. 2024Why Only Text: Empowering Vision-and-Language Navigation with Multi-modal Promptscited by 9
  17. 2022Define, Evaluate, and Improve Task-Oriented Cognitive Capabilities for Instruction Generation Modelscited by 7
  18. 2023Hallucination Detection for Grounded Instruction Generationcited by 7
  19. 2024Can LLMs Generate Human-Like Wayfinding Instructions? Towards Platform-Agnostic Embodied Instruction Synthesiscited by 6
  20. 2024Semantic Map-based Generation of Navigation Instructionscited by 6
  21. 2024MAGIC: Meta-Ability Guided Interactive Chain-of-Distillation for Effective-and-Efficient Vision-and-Language Navigationcited by 6
  22. 2024PRET: Planning with Directed Fidelity Trajectory for Vision and Language Navigationcited by 6
  23. 2024MiniVLN: Efficient Vision-and-Language Navigation by Progressive Knowledge Distillationcited by 6
  24. 2025NavQ: Learning a Q-Model for Foresighted Vision-and-Language Navigationcited by 6
  25. 2025NavComposer: Composing Language Instructions for Navigation Trajectories through Action-Scene-Object Modularizationcited by 5
  26. 2023Masked Path Modeling for Vision-and-Language Navigationcited by 4
  27. 2025Generating Vision-Language Navigation Instructions Incorporated Fine-Grained Alignment Annotationscited by 4
  28. 2025DAgger Diffusion Navigation: DAgger Boosted Diffusion Policy for Vision-Language Navigationcited by 4
  29. 2026Trajectory-Diversity-Driven Robust Vision-and-Language Navigationcited by 4
  30. 2023Which way is `right'?: Uncovering limitations of Vision-and-Language Navigation modelcited by 3
  31. 2025TRAVEL: Training-Free Retrieval and Alignment for Vision-and-Language Navigationcited by 3
  32. 2025DialNav: Multi-turn Dialog Navigation with a Remote Guidecited by 3
  33. 2025Embodied Navigation with Auxiliary Task of Action Description Predictioncited by 3
  34. 2023What Is Near?: Room Locality Learning for Enhanced Robot Vision-Language-Navigation in Indoor Living Environmentscited by 2
  35. 2023Does VLN Pretraining Work with Nonsensical or Irrelevant Instructions?cited by 2
  36. 2024Successfully Guiding Humans with Imperfect Instructions by Highlighting Potential Errors and Suggesting Correctionscited by 2
  37. 2024Continual Vision-and-Language Navigationcited by 2
  38. 2024AIGeN: An Adversarial Approach for Instruction Generation in VLNcited by 2

50 most cited of 62 All 62 (JSON)