mlsanityData to 7 Oct 2026

Semantic Map-based Generation of Navigation Instructions

arXiv 2403.19603 ↗First posted 28 Mar 2024cs.CL cs.AI cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

We are interested in the generation of navigation instructions, either in their own right or as training material for robotic navigation task. In this paper, we propose a new approach to navigation instruction generation by framing the problem as an image captioning task using semantic maps as visual input. Conventional approaches employ a sequence of panorama images to generate navigation instructions. Semantic maps abstract away from visual details and fuse the information in multiple panorama images into a single top-down representation, thereby reducing computational complexity to process the input. We present a benchmark dataset for instruction generation using semantic maps, propose an initial model and ask human subjects to manually assess the quality of generated instructions. Our initial investigations show promise in using semantic maps for instruction generation instead of a sequence of panorama images, but there is vast scope for improvement. We release the code for data preparation and model training at https://github.com/chengzu-li/VLGen.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 56 of its 81 references that are in the map. First posted on arXiv 28 Mar 2024.

Builds on

56 of 81 references.

References · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  11. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  12. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
38 more
  1. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  2. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  3. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  4. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  5. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  6. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  7. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  8. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  9. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  10. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  11. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  12. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  13. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  14. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  15. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  16. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  17. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  18. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  19. 2020Learning to Explore using Active Neural SLAMcited by 308
  20. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  21. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  22. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
  23. 2021PTR: Prompt Tuning with Rules for Text Classificationcited by 238
  24. 2017Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluationcited by 206
  25. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
  26. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 134
  27. 2019Stay on the Path: Instruction Fidelity in Vision-and-Language Navigationcited by 129
  28. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 117
  29. 2018Mapping Instructions to Actions in 3D Environments with Visual Goal Predictioncited by 101
  30. 2021CLIP4Caption: CLIP for Video Captioncited by 88
  31. 2020Sub-Instruction Aware Vision-and-Language Navigationcited by 64
  32. 2021Less is More: Generating Grounded Navigation Instructions from Landmarkscited by 62
  33. 2022A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learningcited by 60
  34. 2022CLIP-Nav: Using CLIP for Zero-Shot Vision-and-Language Navigationcited by 55
  35. 2019Just Ask:An Interactive Learning Framework for Vision and Language Navigationcited by 53
  36. 2022Counterfactual Cycle-Consistent Learning for Instruction Following and Generation in Vision-Language Navigationcited by 50
  37. 2020InterBERT: Vision-and-Language Interaction for Multi-modal Pretrainingcited by 48
  38. 2021On the Evaluation of Vision-and-Language Navigation Instructionscited by 45

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 1.

6 papers.

  1. 2025Visual Planning: Let's Think Only with Imagescited by 85
  2. 2024TopViewRS: Vision-Language Models as Top-View Spatial Reasonerscited by 48
  3. 2025MapTrace: Scalable Data Generation for Route Tracing on Mapscited by 3
  4. 2024NAVCON: A Cognitively Inspired and Linguistically Grounded Corpus for Vision and Language Navigationcited by 0
  5. 2025Lifting Vision: Ground to Aerial Localization with Reasoning Guided Planningcited by 0
  6. 2026NavOne: One-Step Global Planning for Vision-Language Navigation on Top-Down Mapscited by 0