mlsanityData to 7 Oct 2026

Towards Learning a Generalist Model for Embodied Navigation

arXiv 2312.02010 ↗First posted 4 Dec 2023cs.CV cs.AIcited by 135

Not on the map. Why?

Cited by 27 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Building a generalist agent that can interact with the world is the intriguing target of AI systems, thus spurring the research for embodied navigation, where an agent is required to navigate according to instructions or respond to queries. Despite the major progress attained, previous works primarily focus on task-specific agents and lack generalizability to unseen scenarios. Recently, LLMs have presented remarkable capabilities across various fields, and provided a promising opportunity for embodied navigation. Drawing on this, we propose the first generalist model for embodied navigation, NaviLLM. It adapts LLMs to embodied navigation by introducing schema-based instruction. The schema-based instruction flexibly casts various tasks into generation problems, thereby unifying a wide range of tasks. This approach allows us to integrate diverse data sources from various datasets into the training, equipping NaviLLM with a wide range of capabilities required by embodied navigation. We conduct extensive experiments to evaluate the performance and generalizability of our model. The experimental results demonstrate that our unified model achieves state-of-the-art performance on CVDN, SOON, and ScanQA. Specifically, it surpasses the previous stats-of-the-art method by a significant margin of 29% in goal progress on CVDN. Moreover, our model also demonstrates strong generalizability and presents impressive results on unseen tasks, e.g., embodied question answering and 3D captioning.

Key facts. Cited by 135 papers in the map (data through 7 Oct 2026). Builds on 53 of its 57 references that are in the map. First posted on arXiv 4 Dec 2023.

Builds on

53 of 57 references.

References · 53

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  8. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  9. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  10. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  11. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  12. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
38 more
  1. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  2. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  3. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  4. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  5. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  6. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  7. 2023Instruction Tuning with GPT-4cited by 676
  8. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  9. 2018On Evaluation of Embodied Navigation Agentscited by 526
  10. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  11. 2019Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Datasetcited by 426
  12. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  13. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  14. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 338
  15. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  16. 2017Embodied Question Answeringcited by 335
  17. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  18. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  19. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276
  20. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  21. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  22. 2019Vision-and-Dialog Navigationcited by 246
  23. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  24. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 177
  25. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  26. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 129
  27. 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 118
  28. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 117
  29. 2019Embodied Question Answering in Photorealistic Environments with Point Cloud Perceptioncited by 103
  30. 2021Dialogue State Tracking with a Language Model using Schema-Driven Promptingcited by 94
  31. 2022HOP: History-and-Order Aware Pre-training for Vision-and-Language Navigationcited by 69
  32. 2020Sub-Instruction Aware Vision-and-Language Navigationcited by 64
  33. 2022CLIP-Nav: Using CLIP for Zero-Shot Vision-and-Language Navigationcited by 55
  34. 2022Learning from Unlabeled 3D Environments for Vision-and-Language Navigationcited by 49
  35. 2023Improving Vision-and-Language Navigation by Generating Future-View Image Semanticscited by 46
  36. 2018Talk the Walk: Navigating New York City through Grounded Dialoguecited by 44
  37. 2020Environment-agnostic Multitask Learning for Natural Language Grounded Navigationcited by 43
  38. 2023KERM: Knowledge Enhanced Reasoning for Vision-and-Language Navigationcited by 38

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 27; 2025: 59; 2026: 49.

135 papers.

Sort:

By area:

  1. 2025MoGe-2: Accurate Monocular Geometry with Metric Scale and Sharp Detailscited by 250
  2. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
  3. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 201
  4. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  5. 2024Uni-NaVid: A Video-based Vision-Language-Action Model for Unifying Embodied Navigation Taskscited by 146
  6. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understandingcited by 131
  7. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priorscited by 111
  8. 2025StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modelingcited by 107
  9. 2024EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Modelscited by 100
  10. 2024NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Modelscited by 96
  11. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  12. 2024NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoningcited by 77
38 more
  1. 2024Towards Long-Horizon Vision-Language Navigation: Platform, Benchmark and Methodcited by 66
  2. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understandingcited by 63
  3. 2025VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuningcited by 61
  4. 2024Affordances-Oriented Planning using Foundation Models for Continuous Vision-Language Navigationcited by 60
  5. 2024Open-Nav: Exploring Zero-Shot Vision-and-Language Navigation in Continuous Environment with Open-Source LLMscited by 60
  6. 2024Constraint-Aware Zero-Shot Vision-Language Navigation in Continuous Environmentscited by 49
  7. 2025TrackVLA: Embodied Visual Tracking in the Wildcited by 46
  8. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 43
  9. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 41
  10. 2025OctoNav: Towards Generalist Embodied Navigationcited by 33
  11. 2025CityEQA: A Hierarchical LLM Agent on Embodied Question Answering Benchmark in City Spacecited by 31
  12. 20253D Aware Region Prompted Vision Language Modelcited by 31
  13. 2025SmartWay: Enhanced Waypoint Prediction and Backtracking for Zero-Shot Vision-and-Language Navigationcited by 29
  14. 2025Beyond the Destination: A Novel Benchmark for Exploration-Aware Embodied Question Answeringcited by 29
  15. 2024A Survey of Useful LLM Evaluationcited by 28
  16. 2025OST-Bench: Evaluating the Capabilities of MLLMs in Online Spatio-temporal Scene Understandingcited by 24
  17. 2024RoomTour3D: Geometry-Aware Video-Instruction Tuning for Embodied Navigationcited by 23
  18. 2025System-1.5 Reasoning: Traversal in Language and Latent Spaces with Dynamic Shortcutscited by 23
  19. 2024RoboMatrix: A Skill-centric Hierarchical Framework for Scalable Robot Task Planning and Execution in Open-Worldcited by 22
  20. 2025Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigationcited by 21
  21. 2024Vision-Language Navigation with Energy-Based Policycited by 19
  22. 2025Describe Anything Anywhere At Any Momentcited by 19
  23. 2024Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheelcited by 18
  24. 2025NavRAG: Generating User Demand Instructions for Embodied Navigation through Retrieval-Augmented LLMcited by 18
  25. 2024SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Expertscited by 16
  26. 2025A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AIcited by 16
  27. 2025Cross from Left to Right Brain: Adaptive Text Dreamer for Vision-and-Language Navigationcited by 16
  28. 2025Towards Physically Executable 3D Gaussian for Embodied Navigationcited by 16
  29. 2025Unifying 2D and 3D Vision-Language Understandingcited by 15
  30. 2024doScenes: An Autonomous Driving Dataset with Natural Language Instruction for Human Interaction and Vision-Language Navigationcited by 14
  31. 2025Advancing Audio-Visual Navigation Through Multi-Agent Collaboration in 3D Environmentscited by 13
  32. 2025R3Mem: Bridging Memory Retention and Retrieval via Reversible Compressioncited by 12
  33. 2025COSMO: Combination of Selective Memorization for Low-cost Vision-and-Language Navigationcited by 12
  34. 2025Dynamic Multi-Target Fusion for Efficient Audio-Visual Navigationcited by 12
  35. 2024RG-SAN: Rule-Guided Spatial Awareness Network for End-to-End 3D Referring Expression Segmentationcited by 11
  36. 20253D Question Answering via only 2D Vision-Language Modelscited by 10
  37. 2026Action Images: End-to-End Policy Learning via Multiview Video Generationcited by 10
  38. 2025When Robots Should Say "I Don't Know": Benchmarking Abstention in Embodied Question Answeringcited by 9

50 most cited of 135 All 135 (JSON)