mlsanityData to 7 Oct 2026

Think, Act, and Ask: Open-World Interactive Personalized Robot Navigation

arXiv 2310.07968 ↗First posted 12 Oct 2023cs.RO cs.CL cs.HCcited by 31

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

Zero-Shot Object Navigation (ZSON) enables agents to navigate towards open-vocabulary objects in unknown environments. The existing works of ZSON mainly focus on following individual instructions to find generic object classes, neglecting the utilization of natural language interaction and the complexities of identifying user-specific objects. To address these limitations, we introduce Zero-shot Interactive Personalized Object Navigation (ZIPON), where robots need to navigate to personalized goal objects while engaging in conversations with users. To solve ZIPON, we propose a new framework termed Open-woRld Interactive persOnalized Navigation (ORION), which uses Large Language Models (LLMs) to make sequential decisions to manipulate different modules for perception, navigation and communication. Experimental results show that the performance of interactive agents that can leverage user feedback exhibits significant improvement. However, obtaining a good balance between task completion and the efficiency of navigation and interaction remains challenging for all methods. We further provide more findings on the impact of diverse user feedback forms on the agents' performance. Code is available at https://github.com/sled-group/navchat.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 31 of its 53 references that are in the map. First posted on arXiv 12 Oct 2023.

Builds on

31 of 53 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners4 Jul
  2. 2022Inner Monologue: Embodied Reasoning through Planning with Language Models12 Jul

Other references · 29

  1. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  2. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  3. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  4. 2022Language-driven Semantic Segmentationcited by 711
  5. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  6. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  7. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  8. 2022Visual Language Maps for Robot Navigationcited by 362
  9. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  10. 2019Vision-and-Dialog Navigationcited by 246
  11. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  12. 2022ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddingscited by 177
17 more
  1. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 164
  2. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 159
  3. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 134
  4. 2023ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigationcited by 126
  5. 2021GALAXY: A Generative Pre-trained Model for Task-Oriented Dialog with Semi-Supervised Learning and Explicit Policy Injectioncited by 102
  6. 2019Personalized Dialogue Generation with Diversified Traitscited by 73
  7. 2022Correcting Robot Plans with Natural Language Feedbackcited by 68
  8. 2023"No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomycited by 60
  9. 2018Neural User Simulation for Corpus-based Policy Optimisation for Spoken Dialogue Systemscited by 42
  10. 2021Transferable Dialogue Systems and User Simulatorscited by 35
  11. 2022SPACE-3: Unified Dialog Model Pre-training for Task-Oriented Dialog Understanding and Generationcited by 34
  12. 2022SPACE-2: Tree-Structured Semi-Supervised Contrastive Pre-training for Task-Oriented Dialog Understandingcited by 28
  13. 2018Learning Personalized End-to-End Goal-Oriented Dialogcited by 19
  14. 2021Task-Oriented Dialogue System as Natural Language Generationcited by 14
  15. 2021A Framework for Learning to Request Rich and Contextually Useful Information from Humanscited by 12
  16. 2022Learning Depth Vision-Based Personalized Robot Navigation From Dynamic Demonstrations in Virtual Realitycited by 4
  17. 2022Learning Personalized Human-Aware Robot Navigation Using Virtual Reality Demonstrations from a User Studycited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 11; 2026: 7.

31 papers.

Sort:

By area:

  1. 2024Explore until Confident: Efficient Exploration for Embodied Question Answeringcited by 74
  2. 2024AffordanceLLM: Grounding Affordance from Vision Language Modelscited by 55
  3. 2024VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thoughtcited by 55
  4. 2024RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulationcited by 48
  5. 2024RACER: Rich Language-Guided Failure Recovery Policies for Imitation Learningcited by 46
  6. 2024CityNav: A Large-Scale Dataset for Real-World Aerial Navigationcited by 40
  7. 2024VLM See, Robot Do: Human Demo Video to Robot Action Plan via Vision Language Modelcited by 36
  8. 2025A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AIcited by 16
  9. 2026Learning Personalized Agents from Human Feedbackcited by 16
  10. 2025Embodied Agents Meet Personalization: Investigating Challenges and Solutions Through the Lens of Memory Utilizationcited by 15
  11. 2024ReSpAct: Harmonizing Reasoning, Speaking, and Acting Towards Building Large Language Model-Based Conversational AI Agentscited by 14
  12. 2025Memory-Centric Embodied Question Answeringcited by 12
19 more
  1. 2024Personalized Instance-based Navigation Toward User-Specific Objects in Realistic Environmentscited by 11
  2. 2025AmbiK: Dataset of Ambiguous Tasks in Kitchen Environmentcited by 8
  3. 2025CodeDiffuser: Attention-Enhanced Diffusion Policy via VLM-Generated Code for Instruction Ambiguitycited by 8
  4. 2024Tell Me Where You Are: Multimodal LLMs Meet Place Recognitioncited by 7
  5. 2024Teaching Embodied Reinforcement Learning Agents: Informativeness and Diversity of Language Usecited by 7
  6. 2024Collaborative Instance Object Navigation: Leveraging Uncertainty-Awareness to Minimize Human-Agent Dialoguescited by 7
  7. 2024Simulating User Agents for Embodied Conversational-AIcited by 6
  8. 2025CuriousBot: Interactive Mobile Exploration via Actionable 3D Relational Object Graphcited by 4
  9. 2025PersONAL: Towards a Comprehensive Benchmark for Personalized Embodied Agentscited by 4
  10. 2025Human-like Navigation in a World Built for Humanscited by 4
  11. 2026MINT: Minimal Information Neuro-Symbolic Tree for Objective-Driven Knowledge-Gap Reasoning and Active Elicitationcited by 4
  12. 20263D Instruction Ambiguity Detectioncited by 3
  13. 2026Generalized Recognition of Basic Surgical Actions Enables Skill Assessment and Vision-Language-Model-based Surgical Planningcited by 3
  14. 2025Disrupting Vision-Language Model-Driven Navigation Services via Adversarial Object Fusioncited by 2
  15. 2025Toward Ownership Understanding of Objects: Active Question Generation with Large Language Model and Probabilistic Generative Modelcited by 1
  16. 2026Meanings and Measurements: Multi-Agent Probabilistic Grounding for Vision-Language Navigationcited by 1
  17. 2025Freeze and Cluster: A Simple Baseline for Rehearsal-Free Continual Category Discoverycited by 0
  18. 2026Personalizing Embodied Multimodal Large Language Model Agents over Long-term User Interactionscited by 0
  19. 2026Hierarchical Fast-Slow ReAct Agent for Zero-Shot Object-Goal Navigationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigation
  2. 2024Hey Robot! Personalizing Robot Navigation through Model Predictive Control with a Large Language Model
  3. 2025osmAG-LLM: Zero-Shot Open-Vocabulary Object Navigation via Semantic Maps and Large Language Models Reasoning
  4. 2026OpenFrontier: General Navigation with Visual-Language Grounded Frontiers
  5. 2023CorNav: Autonomous Agent with Self-Corrected Planning for Zero-Shot Vision-and-Language Navigation
  6. 2024Zero-shot Object Navigation with Vision-Language Models Reasoning
  7. 2024VLN-Game: Vision-Language Equilibrium Search for Zero-Shot Semantic Navigation
  8. 2024OpenFMNav: Towards Open-Set Zero-Shot Object Navigation via Vision-Language Foundation Models