mlsanityData to 14 Oct 2026

Resolving Positional Ambiguity in Dialogues by Vision-Language Models for Robot Navigation

arXiv 2410.12802 ↗First posted 30 Sep 2024cs.ROcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

We consider an autonomous navigation robot that can accept human commands through natural language to provide services in an indoor environment. These natural language commands may include time, position, object, and action components. However, we observe that the positional components within such commands usually refer to objects in the environment that may contain different levels of positional ambiguity. For example, the command "Go to the chair!" may be ambiguous when there are multiple chairs of the same type in a room. In order to disambiguate these commands, we employ a large language model and a large vision-language model to conduct multiple turns of conversations with the user. We propose a two-level approach that utilizes a vision-language model to map the meanings in natural language to a unique object ID in images and then performs another mapping from the unique object ID to a 3D depth map, thereby allowing the robot to navigate from its current position to the target position. To the best of our knowledge, this is the first work linking foundation models to the positional ambiguity issue.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 16 of its 29 references that are in the map. First posted on arXiv 30 Sep 2024.

Builds on

16 of 29 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Visual Language Maps for Robot Navigation11 Oct

Other references · 15

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023Segment Anythingcited by 8,234
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  7. 2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 503
  8. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  9. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 280
  10. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 271
  11. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 231
  12. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
3 more
  1. 2021Structured Scene Memory for Vision-Language Navigationcited by 95
  2. 2022EnvEdit: Environment Editing for Vision-and-Language Navigationcited by 83
  3. 2021Vision-Language Navigation with Random Environmental Mixupcited by 70

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026AnchorVLN: Geometry-Anchored Vision-Language Grounding Reasoning for Open-Vocabulary Navigation
  2. 2024Integrating Disambiguation and User Preferences into Large Language Models for Robot Motion Planning
  3. 2024IVLMap: Instance-Aware Visual Language Grounding for Consumer Robot Navigation
  4. 2025MLFM: Multi-Layered Feature Maps for Richer Language Understanding in Zero-Shot Semantic Navigation
  5. 2025osmAG-LLM: Zero-Shot Open-Vocabulary Object Navigation via Semantic Maps and Large Language Models Reasoning
  6. 2026A Modular Vision-Language-Action Robotics Framework for Indoor Environments
  7. 2026Sim-to-Real Transfer of Vision-Language Navigation in Continuous Environments Using an Ackermann-Steered Mobile Robot
  8. 2026Meanings and Measurements: Multi-Agent Probabilistic Grounding for Vision-Language Navigation