mlsanityData to 14 Oct 2026

Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planning

arXiv 2310.10103 ↗First posted 16 Oct 2023cs.RO cs.AI cs.CLcited by 89

Not on the map. Why?

Abstract (arXiv)

Navigation in unfamiliar environments presents a major challenge for robots: while mapping and planning techniques can be used to build up a representation of the world, quickly discovering a path to a desired goal in unfamiliar settings with such methods often requires lengthy mapping and exploration. Humans can rapidly navigate new environments, particularly indoor environments that are laid out logically, by leveraging semantics -- e.g., a kitchen often adjoins a living room, an exit sign indicates the way out, and so forth. Language models can provide robots with such knowledge, but directly using language models to instruct a robot how to reach some destination can also be impractical: while language models might produce a narrative about how to reach some goal, because they are not grounded in real-world observations, this narrative might be arbitrarily wrong. Therefore, in this paper we study how the ``semantic guesswork'' produced by language models can be utilized as a guiding heuristic for planning algorithms. Our method, Language Frontier Guide (LFG), uses the language model to bias exploration of novel real-world environments by incorporating the semantic knowledge stored in language models as a search heuristic for planning with either topological or metric maps. We evaluate LFG in challenging real-world environments and simulated benchmarks, outperforming uninformed exploration and other ways of using language models.

Key facts. Cited by 89 papers in the map (data through 14 Oct 2026). Builds on 36 of its 54 references that are in the map. First posted on arXiv 16 Oct 2023.

Builds on

36 of 54 references.

References · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  4. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  5. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  6. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  7. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  8. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  9. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 614
  10. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 611
  11. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  12. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 407
24 more
  1. 2022Visual Language Maps for Robot Navigationcited by 365
  2. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 362
  3. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 304
  4. 2023NoMaD: Goal Masked Diffusion Policies for Navigation and Explorationcited by 248
  5. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 224
  6. 2023ViNT: A Foundation Model for Visual Navigationcited by 222
  7. 2018Semi-parametric Topological Memory for Navigationcited by 209
  8. 2022ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddingscited by 178
  9. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  10. 2022GNM: A General Navigation Model to Drive Any Robotcited by 171
  11. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 165
  12. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 160
  13. 2022PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learningcited by 150
  14. 2023Task and Motion Planning with Large Language Models for Object Rearrangementcited by 141
  15. 2023Translating Natural Language to Planning Goals with Large-Language Modelscited by 133
  16. 2021Rapid Exploration for Open-World Navigation with Latent Goal Modelscited by 121
  17. 2023L3MVN: Leveraging Large Language Models for Visual Target Navigationcited by 120
  18. 2022Navigating to Objects in the Real Worldcited by 106
  19. 2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
  20. 2023Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigationcited by 74
  21. 2023OVRL-V2: A simple state-of-art baseline for ImageNav and ObjectNavcited by 70
  22. 2022ViKiNG: Vision-Based Kilometer-Scale Navigation with Geographic Hintscited by 58
  23. 2019Deep Visual MPC-Policy Learning for Navigationcited by 56
  24. 2020Semantic Curiosity for Active Visual Learningcited by 45

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 47; 2025: 28; 2026: 11.

89 papers.

Sort:

By area:

  1. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 290
  2. 2024Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Modelcited by 268
  3. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 190
  4. 20243D-Mem: 3D Scene Memory for Embodied Exploration and Reasoningcited by 75
  5. 2024Explore until Confident: Efficient Exploration for Embodied Question Answeringcited by 74
  6. 2024Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphscited by 73
  7. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agentscited by 72
  8. 2024A Systematic Survey on Large Language Models for Algorithm Designcited by 69
  9. 2024OpenFMNav: Towards Open-Set Zero-Shot Object Navigation via Vision-Language Foundation Modelscited by 67
  10. 2024ReMEmbR: Building and Reasoning Over Long-Horizon Spatio-Temporal Memory for Robot Navigationcited by 67
  11. 2024VoroNav: Voronoi-based Zero-shot Object Navigation with Large Language Modelcited by 57
  12. 2024Constraint-Aware Zero-Shot Vision-Language Navigation in Continuous Environmentscited by 50
38 more
  1. 2025WMNav: Integrating Vision-Language Models into World Models for Object Goal Navigationcited by 43
  2. 2024On the Prospects of Incorporating Large Language Models (LLMs) in Automated Planning and Scheduling (APS)cited by 40
  3. 2024AgentsCoDriver: Large Language Model Empowered Collaborative Driving with Lifelong Learningcited by 40
  4. 2024GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answeringcited by 37
  5. 2025Safety Guardrails for LLM-Enabled Robotscited by 35
  6. 2024Learning Generalizable Feature Fields for Mobile Manipulationcited by 31
  7. 2025Mem2Ego: Empowering Vision-Language Models with Global-to-Ego Memory for Long-Horizon Embodied Navigationcited by 30
  8. 2025CAST: Counterfactual Labels Improve Instruction Following in Vision-Language-Action Modelscited by 29
  9. 2024Odyssey: Empowering Minecraft Agents with Open-World Skillscited by 26
  10. 2025Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directionscited by 23
  11. 2024SPINE: Online Semantic Planning for Missions with Incomplete Natural Language Specifications in Unstructured Environmentscited by 18
  12. 2024MMRo: Are Multimodal LLMs Eligible as the Brain for In-Home Robotics?cited by 15
  13. 2024Commonsense Reasoning for Legged Robot Adaptation with Vision-Language Modelscited by 15
  14. 2024EfficientEQA: An Efficient Approach to Open-Vocabulary Embodied Question Answeringcited by 15
  15. 2025Enter the Mind Palace: Reasoning and Planning for Long-term Active Embodied Question Answeringcited by 14
  16. 2025Exploring Embodied Multimodal Large Models: Development, Datasets, and Future Directionscited by 13
  17. 2024Tag Map: A Text-Based Map for Spatial Reasoning and Navigation with Large Language Modelscited by 12
  18. 2025NavigateDiff: Visual Predictors are Zero-Shot Navigation Assistantscited by 11
  19. 2025ConformalNL2LTL: Translating Natural Language Instructions into Temporal Logic Formulas with Conformal Correctness Guaranteescited by 11
  20. 2025Air-Ground Collaboration for Language-Specified Missions in Unknown Environmentscited by 11
  21. 2024Guide-LLM: An Embodied LLM Agent and Text-Based Topological Map for Robotic Guidance of People with Visual Impairmentscited by 10
  22. 2025Distilling On-device Language Models for Robot Planning with Minimal Human Interventioncited by 10
  23. 2026VLingNav: Embodied Navigation with Adaptive Reasoning and Visual-Assisted Linguistic Memorycited by 10
  24. 2023Ask more, know better: Reinforce-Learned Prompt Questions for Decision Making with Large Language Modelscited by 9
  25. 2024Multimodal LLM Guided Exploration and Active Mapping using Fisher Informationcited by 9
  26. 2025WoMAP: World Models For Embodied Open-Vocabulary Object Localizationcited by 9
  27. 2024How Secure Are Large Language Models (LLMs) for Navigation in Urban Environments?cited by 8
  28. 2024CAMON: Cooperative Agents for Multi-Object Navigation with LLM-based Conversationscited by 7
  29. 2024Generative AI Agents in Autonomous Machines: A Safety Perspectivecited by 7
  30. 2025A roadmap for AI in roboticscited by 7
  31. 2024OpenDAS: Open-Vocabulary Domain Adaptation for 2D and 3D Segmentationcited by 6
  32. 2024Jailbreaking LLM-Controlled Robotscited by 6
  33. 2024IPPON: Common Sense Guided Informative Path Planning for Object Goal Navigationcited by 6
  34. 2025DORAEMON: Decentralized Ontology-aware Reliable Agent with Enhanced Memory Oriented Navigationcited by 6
  35. 2024Beyond Text: Utilizing Vocal Cues to Improve Decision Making in LLMs for Robot Navigation Taskscited by 5
  36. 2025Mobile Robot Navigation Using Hand-Drawn Maps: A Vision Language Model Approachcited by 5
  37. 2025EfficientNav: Towards On-Device Object-Goal Navigation with Navigation Map Caching and Retrievalcited by 5
  38. 2024DOZE: A Dataset for Open-Vocabulary Zero-Shot Object Navigation in Dynamic Environmentscited by 4

50 most cited of 89 All 89 (JSON)