# Safe Multi-Robot Coordination via VLM-LLM Reasoning and Reachability Analysis

arXiv: https://arxiv.org/abs/2609.27816 · HTML: https://mlsanity.com/paper/2609.27816/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 41 references that are in the map.
- First posted on arXiv 18 Aug 2026.

## Builds on (most cited first, 22 of 22)

- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models](https://mlsanity.com/paper/2307.05973/) (2023)
- [LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action](https://mlsanity.com/paper/2207.04429/) (2022)
- [SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Models](https://mlsanity.com/paper/2309.10062/) (2023)
- [Scalable Multi-Robot Collaboration with Large Language Models: Centralized or Decentralized Systems?](https://mlsanity.com/paper/2309.15943/) (2023)
- [Large Language Models for Robotics: Opportunities, Challenges, and Perspectives](https://mlsanity.com/paper/2401.04334/) (2024)
- [Multimodal Fusion and Vision-Language Models: A Survey for Robot Vision](https://mlsanity.com/paper/2504.02477/) (2025)
- [Provably Safe Reinforcement Learning via Action Projection using Reachability Analysis and Polynomial Zonotopes](https://mlsanity.com/paper/2210.10691/) (2022)
- [Sparse Polynomial Zonotopes: A Novel Set Representation for Reachability Analysis](https://mlsanity.com/paper/1901.01780/) (2019)
- [Data-Driven Reachability Analysis from Noisy Data](https://mlsanity.com/paper/2105.07229/) (2021)
- [Bridging the Gap Between Safety and Real-Time Performance in Receding-Horizon Trajectory Design for Mobile Robots](https://mlsanity.com/paper/1809.06746/) (2018)
- [Safe LLM-Controlled Robots with Formal Guarantees via Reachability Analysis](https://mlsanity.com/paper/2503.03911/) (2025)
- [Resilient Task Allocation in Heterogeneous Multi-Robot Systems](https://mlsanity.com/paper/2009.04593/) (2020)
- [Data-Driven Set-Based Estimation using Matrix Zonotopes with Set Containment Guarantees](https://mlsanity.com/paper/2101.10784/) (2021)
- [Reachability-based Trajectory Design with Neural Implicit Safety Constraints](https://mlsanity.com/paper/2302.07352/) (2023)
- [REFINE: Reachability-based Trajectory Design using Robust Feedback Linearization and Zonotopes](https://mlsanity.com/paper/2211.11997/) (2022)
- [Safe Planning for Articulated Robots Using Reachability-based Obstacle Avoidance With Spheres](https://mlsanity.com/paper/2402.08857/) (2024)
- [Data-Driven Safety Verification using Barrier Certificates and Matrix Zonotopes](https://mlsanity.com/paper/2504.01007/) (2025)

## Abstract (arXiv)

Safe coordination in heterogeneous machine-to-machine (M2M) robotic systems is challenging when robots differ in sensing capabilities, environmental awareness, and motion execution roles. This paper presents a centralized safety-aware M2M framework for cooperative goal-directed navigation in a heterogeneous mobile robot team comprising a vision-capable quadruped and a camera-less robotic vehicle. The objective is to guide both platforms toward a goal region while avoiding static and dynamic obstacles and preventing unsafe inter-robot interactions. Under the principle of shared perception, the vision-capable robot provides semantic environmental awareness through a centralized server over an MQTT broker, enabling the camera-less platform to navigate using this shared scene representation alongside its own odometry, IMU, and state feedback. A vision-language model (VLM) interprets the visual stream, and the extracted semantic data is mapped into conservative metric geometric constraints, including inflated obstacle sets, safe corridors, and goal regions. A large language model (LLM) proposes high-level task allocations, while physical command authority is restricted to a robot-specific zonotope reachability gate. This verification engine propagates independent reachable tubes to evaluate obstacle avoidance, safe-corridor containment, and inter-robot separation predicates before approving commands. Online experiments across clear-path and dynamic-obstacle scenarios show that the pipeline reliably approves safe motion, triggers conservative replanning or holding maneuvers upon constraint violation, and enforces a strict architectural separation between advisory semantic reasoning and formally verified motor execution.