mlsanityData to 7 Oct 2026

Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigation

arXiv 2303.03480 ↗First posted 6 Mar 2023cs.RO cs.AI cs.CLcited by 74

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

We present LGX (Language-guided Exploration), a novel algorithm for Language-Driven Zero-Shot Object Goal Navigation (L-ZSON), where an embodied agent navigates to a uniquely described target object in a previously unseen environment. Our approach makes use of Large Language Models (LLMs) for this task by leveraging the LLM's commonsense reasoning capabilities for making sequential navigational decisions. Simultaneously, we perform generalized target object detection using a pre-trained Vision-Language grounding model. We achieve state-of-the-art zero-shot object navigation results on RoboTHOR with a success rate (SR) improvement of over 27% over the current baseline of the OWL-ViT CLIP on Wheels (OWL CoW). Furthermore, we study the usage of LLMs for robot navigation and present an analysis of various prompting strategies affecting the model output. Finally, we showcase the benefits of our approach via textitreal-world experiments that indicate the superior performance of LGX in detecting and navigating to visually unique objects.

Key facts. Cited by 74 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 34 of its 56 references that are in the map. First posted on arXiv 6 Mar 2023.

Builds on

34 of 56 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents18 Jan

Other references · 33

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  6. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  7. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  8. 2021Grounded Language-Image Pre-trainingcited by 1,147
  9. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  10. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  11. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  12. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
21 more
  1. 2018On Evaluation of Embodied Navigation Agentscited by 526
  2. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  3. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 421
  4. 2022Visual Language Maps for Robot Navigationcited by 362
  5. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  6. 2019Vision-and-Dialog Navigationcited by 246
  7. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 240
  8. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  9. 2020RoboTHOR: An Open Simulation-to-Real Embodied AI Platformcited by 190
  10. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
  11. 2022ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddingscited by 177
  12. 2022PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learningcited by 149
  13. 2022Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detectioncited by 145
  14. 2020Sim-to-Real Transfer for Vision-and-Language Navigationcited by 88
  15. 2022DialFRED: Dialogue-Enabled Agents for Embodied Instruction Followingcited by 63
  16. 2017Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructionscited by 61
  17. 2022CLIP-Nav: Using CLIP for Zero-Shot Vision-and-Language Navigationcited by 55
  18. 2019Improving Grounded Natural Language Understanding through Human-Robot Dialogcited by 25
  19. 2022Object Memory Transformer for Object Goal Navigationcited by 19
  20. 2018Safe Navigation with Human Instructions in Complex Scenescited by 9
  21. 2020Can a Robot Trust You? A DRL-Based Approach to Trust-Driven Human-Guided Navigationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 37; 2025: 19; 2026: 9.

74 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  2. 2023A Comprehensive Overview of Large Language Modelscited by 639
  3. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  4. 2023VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigationcited by 201
  5. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  6. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
  7. 2023Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planningcited by 88
  8. 2024OpenFMNav: Towards Open-Set Zero-Shot Object Navigation via Vision-Language Foundation Modelscited by 66
  9. 2024AgentsCoDriver: Large Language Model Empowered Collaborative Driving with Lifelong Learningcited by 40
  10. 2024BadRobot: Jailbreaking Embodied LLM Agents in the Physical Worldcited by 40
  11. 2025CAST: Counterfactual Labels Improve Instruction Following in Vision-Language-Action Modelscited by 28
  12. 2024TANGO: Training-free Embodied AI Agents for Open-world Taskscited by 26
38 more
  1. 2024An Interactive Agent Foundation Modelcited by 25
  2. 2024MC-GPT: Empowering Vision-and-Language Navigation with Memory Map and Reasoning Chainscited by 25
  3. 2024Perceive, Reflect, and Plan: Designing LLM Agent for Goal-Directed City Navigation without Instructionscited by 25
  4. 2023Can ChatGPT Enable ITS? The Case of Mixed Traffic Control via Reinforcement Learningcited by 18
  5. 2025Towards Embodied Agentic AI: Review and Classification of LLM- and VLM-Driven Robot Autonomy and Interactioncited by 18
  6. 2024Correctable Landmark Discovery via Large Models for Vision-Language Navigationcited by 16
  7. 2024On the Vulnerability of LLM/VLM-Controlled Roboticscited by 15
  8. 2023Reasoning about the Unseen for Efficient Outdoor Object Navigationcited by 13
  9. 2024Tag Map: A Text-Based Map for Spatial Reasoning and Navigation with Large Language Modelscited by 12
  10. 2024Can LLM be a Good Path Planner based on Prompt Engineering? Mitigating the Hallucination for Path Planningcited by 11
  11. 2025From Seeing to Experiencing: Scaling Navigation Foundation Models with Reinforcement Learningcited by 11
  12. 2023CorNav: Autonomous Agent with Self-Corrected Planning for Zero-Shot Vision-and-Language Navigationcited by 10
  13. 2024Guide-LLM: An Embodied LLM Agent and Text-Based Topological Map for Robotic Guidance of People with Visual Impairmentscited by 10
  14. 2024"Don't forget to put the milk back!" Dataset for Enabling Embodied Agents to Detect Anomalous Situationscited by 9
  15. 2024Language-Conditioned Offline RL for Multi-Robot Navigationcited by 9
  16. 2024Multimodal LLM Guided Exploration and Active Mapping using Fisher Informationcited by 9
  17. 2025ForesightNav: Learning Scene Imagination for Efficient Explorationcited by 9
  18. 2024LOC-ZSON: Language-driven Object-Centric Zero-Shot Object Retrieval and Navigationcited by 8
  19. 2025PanoNav: Mapless Zero-Shot Object Navigation with Panoramic Scene Parsing and Dynamic Memorycited by 8
  20. 2024CAMON: Cooperative Agents for Multi-Object Navigation with LLM-based Conversationscited by 7
  21. 2025FlySearch: Exploring how vision-language models explorecited by 7
  22. 2025TANGO: Traversability-Aware Navigation with Local Metric Control for Topological Goalscited by 7
  23. 2024Can LLMs Generate Human-Like Wayfinding Instructions? Towards Platform-Agnostic Embodied Instruction Synthesiscited by 6
  24. 2024Multi-Agent Planning Using Visual Language Modelscited by 6
  25. 2024MARLIN: Multi-Agent Reinforcement Learning Guided by Language-Based Inter-Robot Negotiationcited by 5
  26. 2025HEAL: An Empirical Study on Hallucinations in Embodied Agents Driven by Large Language Modelscited by 5
  27. 2025ObjectReact: Learning Object-Relative Control for Visual Navigationcited by 5
  28. 2025UrbanVerse: Scaling Urban Simulation by Watching City-Tour Videoscited by 5
  29. 2023ROSGPT_Vision: Commanding Robots Using Only Language Models' Promptscited by 4
  30. 2024DOZE: A Dataset for Open-Vocabulary Zero-Shot Object Navigation in Dynamic Environmentscited by 4
  31. 2024Cognitive Planning for Object Goal Navigation using Generative AI Modelscited by 4
  32. 2024KUNPENG: An Embodied Large Model for Intelligent Maritimecited by 4
  33. 2024Find Everything: A General Vision Language Model Approach to Multi-Object Searchcited by 4
  34. 2024Personalized Embodied Navigation for Portable Object Findingcited by 3
  35. 2024Foundation Models to the Rescue: Deadlock Resolution in Connected Multi-Robot Systemscited by 3
  36. 2024Is the House Ready For Sleeptime? Generating and Evaluating Situational Queries for Embodied Question Answeringcited by 3
  37. 2024Spatially-Aware Speaker for Vision-and-Language Navigation Instruction Generationcited by 3
  38. 2026Think Twice, Act Once: Verifier-Guided Action Selection For Embodied Agentscited by 3

50 most cited of 74 All 74 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LaViRA: Language-Vision-Robot Actions Translation for Zero-Shot Vision Language Navigation in Continuous Environments
  2. 2026ViTL: Temporal Logic-Guided Zero-Shot Natural Language Navigation via Vision-Language Models
  3. 2024TINA: Think, Interaction, and Action Framework for Zero-Shot Vision Language Navigation
  4. 2025Vision-and-Language Navigation with Analogical Textual Descriptions in LLMs
  5. 2025Boosting Zero-Shot VLN via Abstract Obstacle Map-Based Waypoint Prediction with TopoGraph-and-VisitInfo-Aware Prompting
  6. 2026Uni-LaViRA: Language-Vision-Robot Actions Translation for Unified Embodied Navigation
  7. 2026VISOR: VIsual Spatial Object Reasoning for Language-driven Object Navigation
  8. 2023A2Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Models