Scaffolding Foundation Models into Physical-World Agents Pushes the Frontier of Long-Horizon Navigation
On the map: LLM agents and tool use → Multimodal and embodied agents
Abstract (arXiv)
Long-horizon physical-world agents must reason over distant goals while grounding decisions in reliable closed-loop behavior. Today's foundation models split these capabilities: vision-language models (VLMs) infer missing information and adapt high-level plans but remain brittle and inefficient at repeated navigation grounding, while navigation foundation models (NFMs) robustly execute semantic goals but operate as bounded episodes without persistent task-level reasoning. We introduce NavMCP, an agentic scaffolding framework that couples a VLM reasoning agent with an NFM executor for long-horizon exploration. The VLM decides what evidence to seek, where to search, and when to stop, while the NFM grounds each semantic sub-goal into closed-loop navigation. Three channels structure their collaboration: intent translates evidence needs into navigation calls, observation converts rollouts into source-grounded trajectory evidence, and memory accumulates findings, negative evidence, and unresolved goals across calls. This design turns isolated navigation rollouts into persistent embodied interaction without retraining either model. On Embodied Question Answering, NavMCP achieves state-of-the-art results on HM-EQA, MT-HM3D, and EXPRESS-Bench. Under matched agent and executor backbones, it outperforms an episodic interface by 14.9 percentage points on HM-EQA. On a Unitree Go2, NavMCP reaches 78.3% success, with its margin over the strongest baseline growing from 10 to 45 points as the task horizon increases. These results demonstrate the potential of scaffolding complementary foundation models into long-horizon physical-world agents.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 41 of its 43 references that are in the map. First posted on arXiv 31 Aug 2026.
Builds on
41 of 43 references.
Closest ancestors · 3
Same direction, newest first
- 2026Qwen-RobotNav Technical Report: A Scalable Navigation Model Designed for an Agentic Navigation System16 Jun
- 2026Towards Long-horizon Embodied Agents with Tool-Aligned Vision-Language-Action Models13 May
- 2025CityEQA: A Hierarchical LLM Agent on Embodied Question Answering Benchmark in City Space18 Feb
Other references · 38
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
- 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
- 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
- 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
- 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
- 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
- 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
- 2022Visual Language Maps for Robot Navigationcited by 362
- 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
- 2017Embodied Question Answeringcited by 335
26 more
- 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
- 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
- 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 279
- 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
- 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
- 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
- 2023VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigationcited by 201
- 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
- 2017IQA: Visual Question Answering in Interactive Environmentscited by 154
- 2024Uni-NaVid: A Video-based Vision-Language-Action Model for Unifying Embodied Navigation Taskscited by 146
- 2023ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigationcited by 126
- 2025StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modelingcited by 107
- 2024NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Modelscited by 96
- 2024Explore until Confident: Efficient Exploration for Embodied Question Answeringcited by 74
- 20243D-Mem: 3D Scene Memory for Embodied Exploration and Reasoningcited by 73
- 2025Embodied Navigation Foundation Modelcited by 65
- 2024GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answeringcited by 37
- 2025Beyond the Destination: A Novel Benchmark for Exploration-Aware Embodied Question Answeringcited by 29
- 2026ABot-N0: Technical Report on the VLA Foundation Model for Versatile Embodied Navigationcited by 23
- 2025BeliefMapNav: 3D Voxel-Based Belief Map for Zero-Shot Object Navigationcited by 18
- 2024SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Expertscited by 16
- 2026ABot-N1: Toward a General Visual Language Navigation Foundation Modelcited by 5
- 2024Multi-LLM QA with Embodied Explorationcited by 3
- 2025Multi-Step Reasoning for Embodied Question Answering via Tool Augmentationcited by 3
- 2026FAST-EQA: Efficient Embodied Question Answering with Global and Local Region Relevancycited by 3
- 2026Memory-Guided View Refinement for Dynamic Human-in-the-loop EQAcited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026NavHarness: Adaptive Goals for Agentic Vision-Language Navigation
- 2026HarnessVLN: Unifying Training-Free Embodied Navigation through an Agent Harness
- 2026NavHarness: Towards Lifelong Embodied Navigation
- 2026Structured World-State Reasoning for Agentic Robotic Search
- 2026ReflectVLN: Training Vision-Language Navigation Agents with Reflective Reasoning
- 2026PreAct-Nav: Agentic Reasoning Before Action for Urban Navigation
- 2025Aux-Think: Exploring Reasoning Strategies for Data-Efficient Vision-Language Navigation
- 2025NavMorph: A Self-Evolving World Model for Vision-and-Language Navigation in Continuous Environments