mlsanityData to 7 Oct 2026

LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action

arXiv 2207.04429 ↗First posted 10 Jul 2022cs.RO cs.AI cs.CLcited by 435

Not on the map. Why?

Cited by 64 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Goal-conditioned policies for robotic navigation can be trained on large, unannotated datasets, providing for good generalization to real-world settings. However, particularly in vision-based settings where specifying goals requires an image, this makes for an unnatural interface. Language provides a more convenient modality for communication with robots, but contemporary methods typically require expensive supervision, in the form of trajectories annotated with language descriptions. We present a system, LM-Nav, for robotic navigation that enjoys the benefits of training on unannotated large datasets of trajectories, while still providing a high-level interface to the user. Instead of utilizing a labeled instruction following dataset, we show that such a system can be constructed entirely out of pre-trained models for navigation (ViNG), image-language association (CLIP), and language modeling (GPT-3), without requiring any fine-tuning or language-annotated robot data. We instantiate LM-Nav on a real-world mobile robot and demonstrate long-horizon navigation through complex, outdoor environments from natural language instructions. For videos of our experiments, code release, and an interactive Colab notebook that runs in your browser, please check out our project page https://sites.google.com/view/lmnav

Key facts. Cited by 435 papers in the map (data through 7 Oct 2026). Builds on 57 of its 77 references that are in the map. First posted on arXiv 10 Jul 2022.

Builds on

57 of 77 references.

References · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2017Mask R-CNNcited by 7,259
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
38 more
  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  2. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  3. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  4. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  5. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  6. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  7. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  8. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  9. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  10. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  11. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  12. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  13. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  14. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  15. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  16. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  17. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  18. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  19. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
  20. 2020Learning to Explore using Active Neural SLAMcited by 308
  21. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
  22. 2018Semi-parametric Topological Memory for Navigationcited by 203
  23. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  24. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  25. 2019Stay on the Path: Instruction Fidelity in Vision-and-Language Navigationcited by 129
  26. 2021Rapid Exploration for Open-World Navigation with Latent Goal Modelscited by 118
  27. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 116
  28. 2022CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailmentcited by 113
  29. 2018Mapping Instructions to Actions in 3D Environments with Visual Goal Predictioncited by 101
  30. 2020Sim-to-Real Transfer for Vision-and-Language Navigationcited by 88
  31. 2020ViNG: Learning Open-World Navigation with Visual Goalscited by 77
  32. 2017MINOS: Multimodal Indoor Simulator for Navigation in Complex Environmentscited by 66
  33. 2019Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill Discoverycited by 65
  34. 2017PRM-RL: Long-range Robotic Navigation Tasks by Combining Reinforcement Learning and Sampling-based Planningcited by 64
  35. 2022ViKiNG: Vision-Based Kilometer-Scale Navigation with Geographic Hintscited by 55
  36. 2019Deep Visual MPC-Policy Learning for Navigationcited by 54
  37. 2015Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequencescited by 50
  38. 2019The StreetLearn Environment and Datasetcited by 42

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 98; 2024: 148; 2025: 112; 2026: 69.

435 papers.

Sort:

By area:

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  2. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  3. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  4. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  5. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  6. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  7. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  8. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  9. 2022Visual Language Maps for Robot Navigationcited by 362
  10. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
  11. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  12. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
38 more
  1. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 288
  2. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  3. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  4. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 253
  5. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
  6. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 219
  7. 2023ViNT: A Foundation Model for Visual Navigationcited by 210
  8. 2024Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigationcited by 195
  9. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 189
  10. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 187
  11. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  12. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  13. 2022GNM: A General Navigation Model to Drive Any Robotcited by 161
  14. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  15. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  16. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 159
  17. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 159
  18. 2024InstructNav: Zero-shot System for Generic Instruction Navigation in Unexplored Environmentcited by 150
  19. 2023Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Gamecited by 148
  20. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
  21. 2023Towards Learning a Generalist Model for Embodied Navigationcited by 135
  22. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 134
  23. 2023Large Language Models for Robotics: A Surveycited by 132
  24. 2023Distilled Feature Fields Enable Few-Shot Language-Guided Manipulationcited by 130
  25. 2023ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigationcited by 126
  26. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 115
  27. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
  28. 2023DetGPT: Detect What You Need via Reasoningcited by 112
  29. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  30. 2023Discuss Before Moving: Visual Language Navigation via Multi-expert Discussionscited by 108
  31. 2024SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigationcited by 106
  32. 2025What Can RL Bring to VLA Generalization? An Empirical Studycited by 102
  33. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  34. 2023Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planningcited by 88
  35. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectivescited by 88
  36. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  37. 2023Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidancecited by 85
  38. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79

50 most cited of 435 All 435 (JSON)