mlsanityData to 7 Oct 2026

Systematic Multi-Agent Vision-and-Language Navigation: Formulation, Benchmark, and Method

arXiv 2609.35965 ↗First posted 28 Sep 2026cs.CV cs.AI cs.ROcited by 0

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Vision-and-Language Navigation (VLN) has largely focused on a single agent following a single instruction, yet many real-world applications require teams of robots to tackle tasks beyond the capabilities of any individual agent. We present Systematic Multi-Agent Vision-and-Language Navigation, providing, to our knowledge, the first systematic formalization of multi-agent VLN as a constrained coordination problem: each mission consists of subtasks carrying dependency and resource constraints (presence locks and holding chains). A verified four-stage crafting pipeline instantiates the task as MAVLN, comprising 11,724 episodes across 145 scenes with teams of up to four agents under three instruction regimes, accompanied by tailored constraint-aware metrics. We further present TRISS, a coordination-ready navigation system coupling an LLM-based subtask scheduler, a shared topological memory that turns each agent's exploration into team knowledge, and a conflict-aware execution mechanism that realizes simultaneous intentions as collision-free routes. Extensive experiments establish TRISS as a comprehensive baseline and reveal substantial room for improvement across scheduling, planning, and execution, highlighting the challenges of coordinating under MAVLN task constraints. Project page: https://xyz9911.github.io/mavln.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 51 of its 57 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

51 of 57 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks31 Oct
  2. 2024EMOS: Embodiment-aware Heterogeneous Multi-robot Operating System with LLM Agents30 Oct
  3. 2024LLaMAR: Long-Horizon Planning for Multi-Agent Robots in Partially Observable Environments14 Jul

Other references · 48

  1. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  2. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  3. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  4. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  5. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  6. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  7. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 231
  8. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
  9. 2023Habitat 3.0: A Co-Habitat for Humans, Avatars and Robotscited by 220
  10. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 201
  11. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
  12. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 165
36 more
  1. 2023SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Modelscited by 151
  2. 2022Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigationcited by 127
  3. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 115
  4. 2023Discuss Before Moving: Visual Language Navigation via Multi-expert Discussionscited by 108
  5. 2025StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modelingcited by 107
  6. 2023ProAgent: Building Proactive Cooperative Agents with Large Language Modelscited by 97
  7. 2020MultiON: Benchmarking Semantic Map Memory using Multi-Object Navigationcited by 95
  8. 2018PRIMAL: Pathfinding via Reinforcement and Imitation Multi-Agent Learningcited by 89
  9. 2025JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigationcited by 80
  10. 2024Towards Long-Horizon Vision-Language Navigation: Platform, Benchmark and Methodcited by 66
  11. 2025Embodied Navigation Foundation Modelcited by 65
  12. 2020A Cordial Sync: Going Beyond Marginal Policies for Multi-Agent Embodied Taskscited by 52
  13. 2024Constraint-Aware Zero-Shot Vision-Language Navigation in Continuous Environmentscited by 49
  14. 2022Iterative Vision-and-Language Navigationcited by 34
  15. 2023March in Chat: Interactive Prompting for Remote Embodied Referring Expressioncited by 34
  16. 2024LaMMA-P: Generalizable Multi-Agent Long-Horizon Task Allocation and Planning with LM-Driven PDDL Plannercited by 31
  17. 2025SmartWay: Enhanced Waypoint Prediction and Backtracking for Zero-Shot Vision-and-Language Navigationcited by 29
  18. 2025Rethinking the Embodied Gap in Vision-and-Language Navigation: A Holistic Study of Physical and Visual Disparitiescited by 24
  19. 2024LiP-LLM: Integrating Linear Programming and dependency graph with Large Language Models for multi-robot task planningcited by 23
  20. 2021Learning Efficient Multi-Agent Cooperative Visual Explorationcited by 19
  21. 2024FLAME: Learning to Navigate with Multimodal LLM in Urban Environmentscited by 19
  22. 2021Collaborative Visual Navigationcited by 18
  23. 2025NavRAG: Generating User Demand Instructions for Embodied Navigation through Retrieval-Augmented LLMcited by 18
  24. 2025VLNVerse: A Benchmark for Vision-Language Navigation with Versatile, Embodied, Realistic Simulation and Evaluationcited by 17
  25. 2024Enhancing Multi-Robot Semantic Navigation Through Multimodal Chain-of-Thought Score Collaborationcited by 14
  26. 2025AeroDuo: Aerial Duo for UAV-based Vision and Language Navigationcited by 13
  27. 2025RoboOS-NeXT: A Unified Memory-based Framework for Lifelong, Scalable, and Robust Multi-Robot Collaborationcited by 9
  28. 2024Planning from Imagination: Episodic Simulation and Episodic Memory for Vision-and-Language Navigationcited by 8
  29. 2024CAMON: Cooperative Agents for Multi-Object Navigation with LLM-based Conversationscited by 7
  30. 2026DeCoNav: Dialog enhanced Long-Horizon Collaborative Vision-Language Navigationcited by 5
  31. 2025Dream to Recall: Imagination-Guided Experience Retrieval for Memory-Persistent Vision-and-Language Navigationcited by 3
  32. 2025Seeing through Uncertainty: Robust Task-Oriented Optimization in Visual Navigationcited by 2
  33. 2026Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planningcited by 2
  34. 2026Does Peer Observation Help? Vision-Sharing Collaboration for Vision-Language Navigationcited by 2
  35. 2026CoNav-UAV: Cooperative Dual-Altitude Aerial Navigation via Stackelberg Learningcited by 2
  36. 2025Beyond Elicitation: Provision-based Prompt Optimization for Knowledge-Intensive Taskscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026AgentVLN: Towards Agentic Vision-and-Language Navigation
  2. 2026Global Commander and Local Operative: A Dual-Agent Framework for Scene Navigation
  3. 2025Breaking Down and Building Up: Mixture of Skill-Based Vision-and-Language Navigation Agents
  4. 2026A Deployable Embodied Vision-Language Navigation System with Hierarchical Cognition and Context-Aware Exploration ↗
  5. 2026GoalVLM: VLM-driven Object Goal Navigation for Multi-Agent System
  6. 2025A Navigation Framework Utilizing Vision-Language Models
  7. 2026HAM-VLN: Harnessing Hierarchical Agentic Memory for Zero-Shot Vision-and-Language Navigation
  8. 2026ReflectVLN: Training Vision-Language Navigation Agents with Reflective Reasoning