mlsanityData to 7 Oct 2026

NavRAG: Generating User Demand Instructions for Embodied Navigation through Retrieval-Augmented LLM

arXiv 2502.11142 ↗First posted 16 Feb 2025cs.AI cs.CL cs.CVcited by 18

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

Vision-and-Language Navigation (VLN) is an essential skill for embodied agents, allowing them to navigate in 3D environments following natural language instructions. High-performance navigation models require a large amount of training data, the high cost of manually annotating data has seriously hindered this field. Therefore, some previous methods translate trajectory videos into step-by-step instructions for expanding data, but such instructions do not match well with users' communication styles that briefly describe destinations or state specific needs. Moreover, local navigation trajectories overlook global context and high-level task planning. To address these issues, we propose NavRAG, a retrieval-augmented generation (RAG) framework that generates user demand instructions for VLN. NavRAG leverages LLM to build a hierarchical scene description tree for 3D scene understanding from global layout to local details, then simulates various user roles with specific demands to retrieve from the scene tree, generating diverse instructions with LLM. We annotate over 2 million navigation instructions across 861 scenes and evaluate the data quality and navigation performance of trained models.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 35 of its 40 references that are in the map. First posted on arXiv 16 Feb 2025.

Builds on

35 of 40 references.

References · 35

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  4. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,411
  5. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  6. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  7. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  8. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  9. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  10. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  11. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  12. 2023Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergycited by 343
23 more
  1. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  2. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  3. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  4. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  5. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  6. 2021Large Scale Image Completion via Co-Modulated Generative Adversarial Networkscited by 219
  7. 2023Towards Learning a Generalist Model for Embodied Navigationcited by 135
  8. 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 118
  9. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 117
  10. 2023Bird's-Eye-View Scene Graph for Vision-Language Navigationcited by 71
  11. 2021Less is More: Generating Grounded Navigation Instructions from Landmarkscited by 62
  12. 2022A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learningcited by 60
  13. 2024Vision-and-Language Navigation via Causal Learningcited by 51
  14. 2023Learning Vision-and-Language Navigation from YouTube Videoscited by 47
  15. 2024Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generationcited by 44
  16. 2023Lana: A Language-Capable Navigator for Instruction Following and Generationcited by 39
  17. 2023March in Chat: Interactive Prompting for Remote Embodied Referring Expressioncited by 34
  18. 2022Simple and Effective Synthesis of Indoor 3D Scenescited by 33
  19. 2023VLN-Trans: Translator for the Vision and Language Navigation Agentcited by 25
  20. 2024EmbodiedRAG: Dynamic 3D Scene Graph Retrieval for Efficient and Scalable Robot Task Planningcited by 23
  21. 2024Controllable Navigation Instruction Generation with Chain of Thought Promptingcited by 20
  22. 2024Vision-Language Navigation with Energy-Based Policycited by 19
  23. 2023Kefa: A Knowledge Enhanced and Fine-grained Aligned Speaker for Navigation Instruction Generationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 10.

18 papers.

Sort:

By area:

  1. 2025Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigationcited by 21
  2. 2025A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AIcited by 16
  3. 2025Meta-Memory: Retrieving and Integrating Semantic-Spatial Memories for Robot Spatial Reasoningcited by 9
  4. 2025GoViG: Goal-Conditioned Visual Navigation Instruction Generation via Multimodal Reasoningcited by 6
  5. 2025D3D-VLP: Dynamic 3D Vision-Language-Planning Model for Embodied Grounding and Navigationcited by 5
  6. 2026Persona-aware and Explainable Bikeability Assessment: A Vision-Language Model Approachcited by 4
  7. 2025Affordance RAG: Hierarchical Multimodal Retrieval with Affordance-Aware Embodied Memory for Mobile Manipulationcited by 3
  8. 2025FreeAskWorld: An Interactive and Closed-Loop Simulator for Human-Centric Embodied AIcited by 1
  9. 2026TrajRAG: Retrieving Geometric-Semantic Experience for Zero-Shot Object Navigationcited by 1
  10. 2026Image2Sim: Scaling Embodied Navigation via Generative Neural Simulatorcited by 1
  11. 2025Exploring the Link Between Bayesian Inference and Embodied Intelligence: Toward Open Physical-World Embodied AI Systemscited by 0
  12. 2026GA-VLN: Geometry-Aware BEV Representation for Efficient Vision-Language Navigationcited by 0
6 more
  1. 2026Beyond Waypoints: A Trajectory-Centric Waypointing Paradigm for Vision-Language Navigationcited by 0
  2. 2026Advancing DialNav through Automatic Embodied Dialog Augmentationcited by 0
  3. 2026NavVerse: Benchmarking Indoor-to-Outdoor Embodied Navigation in Continuous Robot Simulationcited by 0
  4. 2026Goal-oriented Navigation Instruction Generation with Tour Video Priorscited by 0
  5. 2026ExploreAI: Agentic Exploration Knowledge Bases for Reproducible Observable-Regression Testing of Black-Box VR and 3D Applicationscited by 0
  6. 2026Systematic Multi-Agent Vision-and-Language Navigation: Formulation, Benchmark, and Methodcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Uni-NaVid: A Video-based Vision-Language-Action Model for Unifying Embodied Navigation Tasks
  2. 2025NavBench: Probing Multimodal Large Language Models for Embodied Navigation
  3. 2026NavGen: Visual Generative Models as a Scalable Data Engine for Embodied 3D Navigation
  4. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigation
  5. 2025FiLM-Nav: Efficient and Generalizable Navigation via VLM Fine-tuning
  6. 2025Embodied Navigation Foundation Model
  7. 2026Learning to Retrieve Navigable Candidates for Efficient Vision-and-Language Navigation
  8. 2025NavA3: Understanding Any Instruction, Navigating Anywhere, Finding Anything