mlsanityData to 7 Oct 2026

ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigation

arXiv 2301.13166 ↗First posted 30 Jan 2023cs.AI cs.CV cs.LGcited by 126

Not on the map. Why?

Abstract (arXiv)

The ability to accurately locate and navigate to a specific object is a crucial capability for embodied agents that operate in the real world and interact with objects to complete tasks. Such object navigation tasks usually require large-scale training in visual environments with labeled objects, which generalizes poorly to novel objects in unknown environments. In this work, we present a novel zero-shot object navigation method, Exploration with Soft Commonsense constraints (ESC), that transfers commonsense knowledge in pre-trained models to open-world object navigation without any navigation experience nor any other training on the visual environments. First, ESC leverages a pre-trained vision and language model for open-world prompt-based grounding and a pre-trained commonsense language model for room and object reasoning. Then ESC converts commonsense knowledge into navigation actions by modeling it as soft logic predicates for efficient exploration. Extensive experiments on MP3D, HM3D, and RoboTHOR benchmarks show that our ESC method improves significantly over baselines, and achieves new state-of-the-art results for zero-shot object navigation (e.g., 288% relative Success Rate improvement than CoW on MP3D).

Key facts. Cited by 126 papers in the map (data through 7 Oct 2026). Builds on 41 of its 51 references that are in the map. First posted on arXiv 30 Jan 2023.

Builds on

41 of 51 references.

References · 41

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Mask R-CNNcited by 7,259
  5. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  8. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  9. 2021Grounded Language-Image Pre-trainingcited by 1,147
  10. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  11. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  12. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
29 more
  1. 2018On Evaluation of Embodied Navigation Agentscited by 526
  2. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  3. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 421
  4. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  5. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  6. 2020Learning to Explore using Active Neural SLAMcited by 308
  7. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 240
  8. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  9. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  10. 2020RoboTHOR: An Open Simulation-to-Real Embodied AI Platformcited by 190
  11. 2022ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddingscited by 177
  12. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 159
  13. 2022PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learningcited by 149
  14. 2018Visual Semantic Navigation using Scene Priorscited by 127
  15. 2021FILM: Following Instructions in Language with Modular Methodscited by 119
  16. 2022Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scalecited by 116
  17. 2021Skill Induction and Planning with Latent Languagecited by 99
  18. 2021A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Executioncited by 91
  19. 2022Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigationcited by 86
  20. 2021Auxiliary Tasks and Exploration Enable ObjectNavcited by 73
  21. 2022Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigationcited by 66
  22. 2021Memory-Augmented Reinforcement Learning for Image-Goal Navigationcited by 64
  23. 2015Hinge-Loss Markov Random Fields and Probabilistic Soft Logiccited by 63
  24. 2022UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Trainingcited by 57
  25. 2022TIDEE: Tidying Up Novel Rooms using Visuo-Semantic Commonsense Priorscited by 30
  26. 2022JARVIS: A Neuro-Symbolic Commonsense Reasoning Framework for Conversational Embodied Agentscited by 29
  27. 2020Integrating Egocentric Localization for More Realistic Point-Goal Navigation Agentscited by 27
  28. 2022Learning Active Camera for Multi-Object Navigationcited by 20
  29. 2020Semantic Linking Maps for Active Visual Object Searchcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 40; 2025: 37; 2026: 37.

126 papers.

Sort:

By area:

  1. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  2. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
  3. 2023VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigationcited by 201
  4. 2024InstructNav: Zero-shot System for Generic Instruction Navigation in Unexplored Environmentcited by 150
  5. 2025StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modelingcited by 107
  6. 2024SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigationcited by 106
  7. 2024Explore until Confident: Efficient Exploration for Embodied Question Answeringcited by 74
  8. 2024Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphscited by 72
  9. 2023VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street Viewcited by 67
  10. 2024OpenFMNav: Towards Open-Set Zero-Shot Object Navigation via Vision-Language Foundation Modelscited by 66
  11. 2025UniGoal: Towards Universal Zero-shot Goal-oriented Navigationcited by 56
  12. 2024VoroNav: Voronoi-based Zero-shot Object Navigation with Large Language Modelcited by 55
38 more
  1. 2024RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesiscited by 55
  2. 2024Constraint-Aware Zero-Shot Vision-Language Navigation in Continuous Environmentscited by 49
  3. 2024RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulationcited by 48
  4. 2023LangNav: Language as a Perceptual Representation for Navigationcited by 47
  5. 2025WMNav: Integrating Vision-Language Models into World Models for Object Goal Navigationcited by 42
  6. 2024GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answeringcited by 37
  7. 2024CogNav: Cognitive Process Modeling for Object Goal Navigation with LLMscited by 32
  8. 2023Think, Act, and Ask: Open-World Interactive Personalized Robot Navigationcited by 31
  9. 2024Prioritized Semantic Learning for Zero-shot Instance Navigationcited by 30
  10. 2025Mem2Ego: Empowering Vision-Language Models with Global-to-Ego Memory for Long-Horizon Embodied Navigationcited by 30
  11. 2024MC-GPT: Empowering Vision-and-Language Navigation with Memory Map and Reasoning Chainscited by 25
  12. 2025NavBench: Probing Multimodal Large Language Models for Embodied Navigationcited by 24
  13. 2023ThinkBot: Embodied Instruction Following with Thought Chain Reasoningcited by 19
  14. 2024Zero-shot Object Navigation with Vision-Language Models Reasoningcited by 18
  15. 2024NavAgent: Multi-scale Urban Street View Fusion For UAV Embodied Vision-and-Language Navigationcited by 18
  16. 2024g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Taskscited by 18
  17. 2024Correctable Landmark Discovery via Large Models for Vision-Language Navigationcited by 16
  18. 2024SPINE: Online Semantic Planning for Missions with Incomplete Natural Language Specifications in Unstructured Environmentscited by 16
  19. 2025FlexVLN: Flexible Adaptation for Diverse Vision-and-Language Navigation Taskscited by 16
  20. 2024TopV-Nav: Unlocking the Top-View Spatial Reasoning Potential of MLLM for Zero-shot Object Navigationcited by 15
  21. 2023Advances in Embodied Navigation Using Large Language Models: A Surveycited by 14
  22. 2024EditRoom: LLM-parameterized Graph Diffusion for Composable 3D Room Layout Editingcited by 14
  23. 2023Reasoning about the Unseen for Efficient Outdoor Object Navigationcited by 13
  24. 2024One Map to Find Them All: Real-time Open-Vocabulary Mapping for Zero-shot Multi-Object Navigationcited by 13
  25. 2024A Survey on Human-Centric LLMscited by 13
  26. 2025MSGNav: Unleashing the Power of Multi-modal 3D Scene Graph for Zero-Shot Embodied Navigationcited by 13
  27. 2023Learning Hierarchical Interactive Multi-Object Search for Mobile Manipulationcited by 12
  28. 2024Tag Map: A Text-Based Map for Spatial Reasoning and Navigation with Large Language Modelscited by 12
  29. 2025NavigateDiff: Visual Predictors are Zero-Shot Navigation Assistantscited by 11
  30. 2023CorNav: Autonomous Agent with Self-Corrected Planning for Zero-Shot Vision-and-Language Navigationcited by 10
  31. 2025GC-VLN: Instruction as Graph Constraints for Training-free Vision-and-Language Navigationcited by 10
  32. 2025ForesightNav: Learning Scene Imagination for Efficient Explorationcited by 9
  33. 2025WoMAP: World Models For Embodied Open-Vocabulary Object Localizationcited by 8
  34. 2025PanoNav: Mapless Zero-Shot Object Navigation with Panoramic Scene Parsing and Dynamic Memorycited by 8
  35. 2024How Secure Are Large Language Models (LLMs) for Navigation in Urban Environments?cited by 7
  36. 2024CAMON: Cooperative Agents for Multi-Object Navigation with LLM-based Conversationscited by 7
  37. 2024Collaborative Instance Object Navigation: Leveraging Uncertainty-Awareness to Minimize Human-Agent Dialoguescited by 7
  38. 2023"What's important here?": Opportunities and Challenges of Using LLMs in Retrieving Information from Web Interfacescited by 6

50 most cited of 126 All 126 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigation
  2. 2025FiLM-Nav: Efficient and Generalizable Navigation via VLM Fine-tuning
  3. 2026OpenFrontier: General Navigation with Visual-Language Grounded Frontiers
  4. 2026SkillNav: Score-Level Skill Intervention for Zero-Shot Object Goal Navigation
  5. 2025Imagine, Verify, Execute: Memory-guided Agentic Exploration with Vision-Language Models
  6. 2026IMPRINT: Image-Conditioned Query Enrichment for Long-Tail Object Goal Navigation
  7. 2025VLN-Zero: Rapid Exploration and Cache-Enabled Neurosymbolic Vision-Language Planning for Zero-Shot Transfer in Robot Navigation
  8. 2025Nav-R2 Dual-Relation Reasoning for Generalizable Open-Vocabulary Object-Goal Navigation