Ask When It Pays: Cost-Aware Open-Ended Interaction for Instance Goal Navigation
Not on the map. Why?
Abstract (arXiv)
Instance Goal Navigation (IGN) requires an embodied agent to find a specific object instance among distractors from an under-specified natural-language description. Such ambiguity often cannot be resolved from perception and language alone, making interaction with an oracle a natural mechanism for disambiguation. Prior interactive methods allow oracle queries but treat lightweight clarification and route-level guidance alike, letting agents boost success rate through repeated high-information questions rather than by resolving the underlying ambiguity efficiently. We recast interactive IGN as a cost-sensitive uncertainty-reduction problem, where the agent should ask the question whose answer provides the largest reduction in navigation uncertainty relative to its penalty. To this end, we apply an information-gain analysis on existing navigation corpora to identify which cues reduce navigation uncertainty, yielding a compact set of question types and data-derived weights. However, existing interactive navigation benchmarks do not model the cost of different question types or evaluate how efficiently agents use interaction, making them unsuitable for studying cost-sensitive interaction. Based on this taxonomy, we construct a benchmark for diagnosing interaction behavior and efficiency, together with a Weighted Success Rate metric that penalizes each query by its derived cost. We further propose a zero-shot MLLM navigator that selectively queries at each decision step only when the expected uncertainty reduction justifies the interaction cost.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 59 of its 68 references that are in the map. First posted on arXiv 2 Jun 2026.
Builds on
59 of 68 references.
References · 59
- 2025Qwen3-VL Technical Reportcited by 2,534
- 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
- 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
- 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
- 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
- 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
- 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
- 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
- 2018On Evaluation of Embodied Navigation Agentscited by 526
- 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
- 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
- 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
38 more
- 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
- 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
- 2019Vision-and-Dialog Navigationcited by 246
- 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 240
- 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
- 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 201
- 2021TEACh: Task-driven Embodied Agents that Chatcited by 194
- 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
- 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
- 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 165
- 2024InstructNav: Zero-shot System for Generic Instruction Navigation in Unexplored Environmentcited by 150
- 2023Towards Learning a Generalist Model for Embodied Navigationcited by 135
- 2022Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigationcited by 127
- 2021Waypoint Models for Instruction-guided Navigation in Continuous Environmentscited by 125
- 2023GridMM: Grid Memory Map for Vision-and-Language Navigationcited by 117
- 2023Scaling Data Generation in Vision-and-Language Navigationcited by 115
- 2023Discuss Before Moving: Visual Language Navigation via Multi-expert Discussionscited by 108
- 2019The Regretful Agent: Heuristic-Aided Navigation through Progress Estimationcited by 104
- 2024NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Modelscited by 96
- 2018Vision-based Navigation with Language-based Assistance via Imitation Learning with Indirect Interventioncited by 77
- 2022DialFRED: Dialogue-Enabled Agents for Embodied Instruction Followingcited by 63
- 2024Open-Nav: Exploring Zero-Shot Vision-and-Language Navigation in Continuous Environment with Open-Source LLMscited by 60
- 2025Ground Slow, Move Fast: A Dual-System Foundation Model for Generalizable Vision-and-Language Navigationcited by 56
- 2025MapNav: A Novel Memory Representation via Annotated Semantic Maps for Vision-and-Language Navigationcited by 55
- 2019Just Ask:An Interactive Learning Framework for Vision and Language Navigationcited by 53
- 2024Vision-and-Language Navigation via Causal Learningcited by 51
- 2022Learning from Unlabeled 3D Environments for Vision-and-Language Navigationcited by 49
- 2023LangNav: Language as a Perceptual Representation for Navigationcited by 47
- 2023March in Chat: Interactive Prompting for Remote Embodied Referring Expressioncited by 34
- 2020RMM: A Recursive Mental Model for Dialog Navigationcited by 32
- 2025SmartWay: Enhanced Waypoint Prediction and Backtracking for Zero-Shot Vision-and-Language Navigationcited by 29
- 2020The RobotSlang Benchmark: Dialog-guided Robot Localization and Navigationcited by 26
- 2025NavBench: Probing Multimodal Large Language Models for Embodied Navigationcited by 24
- 2025Rethinking the Embodied Gap in Vision-and-Language Navigation: A Holistic Study of Physical and Visual Disparitiescited by 24
- 2025BED-LLM: Intelligent Information Gathering with LLMs and Bayesian Experimental Designcited by 21
- 2025VLNVerse: A Benchmark for Vision-Language Navigation with Versatile, Embodied, Realistic Simulation and Evaluationcited by 17
- 2024SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Expertscited by 16
- 2025Ground-level Viewpoint Vision-and-Language Navigation in Continuous Environmentscited by 10
50 most cited of 59 All 59 (JSON)
Built on by
Citing papers per year: 2026: 2.
2 papers.