# NavGPT-3: Harnessing Context in a Hierarchical Navigation Runtime arXiv: https://arxiv.org/abs/2610.10787 ยท HTML: https://mlsanity.com/paper/2610.10787/ ## Key facts - No citing papers in the map yet (data through 7 Oct 2026). - Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. - Builds on 36 of its 61 references that are in the map. - First posted on arXiv 7 Oct 2026. ## Placed in - LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/) ## Builds on (most cited first, 30 of 36) - [Qwen-Image Technical Report](https://mlsanity.com/paper/2508.02324/) (2025) - [Moshi: a speech-text foundation model for real-time dialogue](https://mlsanity.com/paper/2410.00037/) (2024) - [ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects](https://mlsanity.com/paper/2006.13171/) (2020) - [NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigation](https://mlsanity.com/paper/2402.15852/) (2024) - [Discuss Before Moving: Visual Language Navigation via Multi-expert Discussions](https://mlsanity.com/paper/2309.11382/) (2023) - [StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modeling](https://mlsanity.com/paper/2507.05240/) (2025) - [MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigation](https://mlsanity.com/paper/2401.07314/) (2024) - [Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphs](https://mlsanity.com/paper/2407.07775/) (2024) - [Embodied Navigation Foundation Model](https://mlsanity.com/paper/2509.12129/) (2025) - [Open-Nav: Exploring Zero-Shot Vision-and-Language Navigation in Continuous Environment with Open-Source LLMs](https://mlsanity.com/paper/2409.18794/) (2024) - [Ground Slow, Move Fast: A Dual-System Foundation Model for Generalizable Vision-and-Language Navigation](https://mlsanity.com/paper/2512.08186/) (2025) - [LangNav: Language as a Perceptual Representation for Navigation](https://mlsanity.com/paper/2310.07889/) (2023) - [GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answering](https://mlsanity.com/paper/2412.14480/) (2024) - [VLA-OS: Structuring and Dissecting Planning Representations and Paradigms in Vision-Language-Action Models](https://mlsanity.com/paper/2506.17561/) (2025) - [OmniNav: A Unified Framework for Prospective Exploration and Visual-Language Navigation](https://mlsanity.com/paper/2509.25687/) (2025) - [LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligence](https://mlsanity.com/paper/2605.25979/) (2026) - [RoboClaw: An Agentic Framework for Scalable Long-Horizon Robotic Tasks](https://mlsanity.com/paper/2603.11558/) (2026) - [ABot-N0: Technical Report on the VLA Foundation Model for Versatile Embodied Navigation](https://mlsanity.com/paper/2602.11598/) (2026) - [LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding](https://mlsanity.com/paper/2605.27365/) (2026) - [AI Harness Engineering: A Runtime Substrate for Foundation-Model Software Agents](https://mlsanity.com/paper/2605.13357/) (2026) - [VLNVerse: A Benchmark for Vision-Language Navigation with Versatile, Embodied, Realistic Simulation and Evaluation](https://mlsanity.com/paper/2512.19021/) (2025) - [AstraNav-World: World Model for Foresight Control and Consistency](https://mlsanity.com/paper/2512.21714/) (2025) - [Memory-Centric Embodied Question Answering](https://mlsanity.com/paper/2505.13948/) (2025) - [OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence](https://mlsanity.com/paper/2602.08683/) (2026) - [Qwen-RobotNav Technical Report: A Scalable Navigation Model Designed for an Agentic Navigation System](https://mlsanity.com/paper/2606.18112/) (2026) - [What Matters in Orchestrating Robot Policies: A Systematic Study of Hierarchical VLA Agents](https://mlsanity.com/paper/2606.10267/) (2026) - [DuplexOmni: Real-Time Listening, Seeing, Thinking, and Speaking for Full-Duplex Interaction](https://mlsanity.com/paper/2606.09186/) (2026) - [HiVLA: A Visual-Grounded-Centric Hierarchical Embodied Manipulation System](https://mlsanity.com/paper/2604.14125/) (2026) - [PLanAR: Planning-Language-Grounded Agentic Reasoning for Robot Manipulation](https://mlsanity.com/paper/2602.01662/) (2026) - [Embodied Agents Take Control: Minimal-Interface Zero-Shot Agents Rival Industrial-Scale Policies in Vision-and-Language Navigation](https://mlsanity.com/paper/2607.26148/) (2026) ## Abstract (arXiv) Language models trained with long-horizon agentic reinforcement learning can generalize knowledge through reasoning, express precise actions, and pursue goals over many steps, raising the ceiling on what an embodied agent can understand and decide. Physical interaction, however, remains the domain of action policies, which provide dense, low-latency control. We present NavGPT-3, a harness that connects the two models, with an OS-like runtime built above it: reasoning, acting, and monitoring run as threads with their own context, tools, and permissions, while the runtime schedules them and decides which thread controls the robot's motion, so that the robot can react to sudden real-world events through interruption and thread switching. Beneath it, our action policy NavGPT VLA, trained on 19.28M examples, allocates visual tokens using codec allocation, in proportion to scene change; its 8B model alone reaches 74.51 SR on R2R-CE and leads RxR-CE with 78.19 SR. With the complete harness, NavGPT-3 sets the state of the art on R2R-CE (81.51 SR) and, for the first time, brings an autonomous agent to human level: on RxR-CE it matches human followers in success (90.43 vs. 90.4 SR) and path fidelity (78.47 vs. 77.7 nDTW) at 1 min 22 s per episode, versus roughly 3 min for a human. We comprehensively ablate the harness design and the interaction between the two models, showing how tools and the action policy shape the path from language-model reasoning to physical control: when NavGPT VLA executes the route, the reasoning loop shortens and the system's minimum reaction time falls from 3-19 s per language-model decision to 0.5-1 s per action-policy step (1-2 Hz). These results show that designing this embodied interface is central to connecting frontier language-model intelligence with low-level physical control. We will release all models, code, and evaluation records.