# HaltNav: Reactive Visual Halting over Lightweight Topological Priors for Robust Vision-Language Navigation

arXiv: https://arxiv.org/abs/2603.12696 · HTML: https://mlsanity.com/paper/2603.12696/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 33 references that are in the map.
- First posted on arXiv 13 Mar 2026.

## Builds on (most cited first, 27 of 27)

- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI](https://mlsanity.com/paper/2109.08238/) (2021)
- [Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments](https://mlsanity.com/paper/2004.02857/) (2020)
- [Visual Language Maps for Robot Navigation](https://mlsanity.com/paper/2210.05714/) (2022)
- [SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning](https://mlsanity.com/paper/2307.06135/) (2023)
- [NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigation](https://mlsanity.com/paper/2402.15852/) (2024)
- [Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Models](https://mlsanity.com/paper/2502.19417/) (2025)
- [Uni-NaVid: A Video-based Vision-Language-Action Model for Unifying Embodied Navigation Tasks](https://mlsanity.com/paper/2412.06224/) (2024)
- [StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modeling](https://mlsanity.com/paper/2507.05240/) (2025)
- [Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detection](https://mlsanity.com/paper/2412.04455/) (2024)
- [Ground Slow, Move Fast: A Dual-System Foundation Model for Generalizable Vision-and-Language Navigation](https://mlsanity.com/paper/2512.08186/) (2025)
- [MapNav: A Novel Memory Representation via Annotated Semantic Maps for Vision-and-Language Navigation](https://mlsanity.com/paper/2502.13451/) (2025)
- [From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons](https://mlsanity.com/paper/2412.08442/) (2024)
- [VLA-OS: Structuring and Dissecting Planning Representations and Paradigms in Vision-Language-Action Models](https://mlsanity.com/paper/2506.17561/) (2025)
- [OmniNav: A Unified Framework for Prospective Exploration and Visual-Language Navigation](https://mlsanity.com/paper/2509.25687/) (2025)
- [Diffusion Trajectory-guided Policy for Long-horizon Robot Manipulation](https://mlsanity.com/paper/2502.10040/) (2025)
- [FSR-VLN: Fast and Slow Reasoning for Vision-Language Navigation with Hierarchical Multi-modal Scene Graph](https://mlsanity.com/paper/2509.13733/) (2025)
- [Guide-LLM: An Embodied LLM Agent and Text-Based Topological Map for Robotic Guidance of People with Visual Impairments](https://mlsanity.com/paper/2410.20666/) (2024)
- [OpenIN: Open-Vocabulary Instance-Oriented Navigation in Dynamic Domestic Environments](https://mlsanity.com/paper/2501.04279/) (2025)
- [osmAG-LLM: Zero-Shot Open-Vocabulary Object Navigation via Semantic Maps and Large Language Models Reasoning](https://mlsanity.com/paper/2507.12753/) (2025)
- [UNeMo: Collaborative Visual-Language Reasoning and Navigation via a Multimodal World Model](https://mlsanity.com/paper/2511.18845/) (2025)
- [DV-VLN: Dual Verification for Reliable LLM-Based Vision-and-Language Navigation](https://mlsanity.com/paper/2601.18492/) (2026)
- [Area Graph: Generation of Topological Maps using the Voronoi Diagram](https://mlsanity.com/paper/1910.01019/) (2019)
- [osmAG: Hierarchical Semantic Topometric Area Graph Maps in the OSM Format for Mobile Robotics](https://mlsanity.com/paper/2309.04791/) (2023)
- [Generation of Indoor Open Street Maps for Robot Navigation from CAD Files](https://mlsanity.com/paper/2507.00552/) (2025)
- [Empowering Robot Path Planning with Large Language Models: osmAG Map Topology & Hierarchy Comprehension with LLMs](https://mlsanity.com/paper/2403.08228/) (2024)

## Abstract (arXiv)

Vision-and-Language Navigation (VLN) is shifting from rigid, step-by-step instruction following toward open-vocabulary, goal-oriented autonomy. Achieving this transition without exhaustive routing prompts requires agents to leverage structural priors. While prior work often assumes computationally heavy 2D/3D metric maps, we instead exploit a lightweight, text-based osmAG (OpenStreetMap Area Graph), a floorplan-level topological representation that is easy to obtain and maintain. However, global planning over a prior map alone is brittle in real-world deployments, where local connectivity can change (e.g., closed doors or crowded passages), leading to execution-time failures. To address this gap, we propose a hierarchical navigation framework HaltNav that couples the robust global planning of osmAG with the local exploration and instruction-grounding capability of VLN. Our approach features an MLLM-based brain module, which is capable of high-level task grounding and obstruction awareness. Conditioned on osmAG, the brain converts the global route into a sequence of localized execution snippets, providing the VLN executor with prior-grounded, goal-centric sub-instructions. Meanwhile, it detects local anomalies via a mechanism we term Reactive Visual Halting (RVH), which interrupts the local control loop, updates osmAG by invalidating the corresponding topology, and triggers replanning to orchestrate a viable detour. To train this halting capability efficiently, we introduce a data synthesis pipeline that leverages generative models to inject realistic obstacles into otherwise navigable scenes, substantially enriching hard negative samples. Extensive experiments demonstrate that our hierarchical framework outperforms several baseline methods without tedious language instructions, and significantly improves robustness for long-horizon vision-language navigation under environmental changes.