# Safe and Interpretable Multimodal Path Planning for Multi-Agent Cooperation arXiv: https://arxiv.org/abs/2602.19304 ยท HTML: https://mlsanity.com/paper/2602.19304/ ## Key facts - Cited by 3 papers in the map (data through 14 Oct 2026). - Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. - Builds on 39 of its 58 references that are in the map. - First posted on arXiv 22 Feb 2026. ## Placed in - LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/) ## Builds on (most cited first, 30 of 39) - [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022) - [Eureka: Human-Level Reward Design via Coding Large Language Models](https://mlsanity.com/paper/2310.12931/) (2023) - [LLM+P: Empowering Large Language Models with Optimal Planning Proficiency](https://mlsanity.com/paper/2304.11477/) (2023) - [VirtualHome: Simulating Household Activities via Programs](https://mlsanity.com/paper/1806.07011/) (2018) - [RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics](https://mlsanity.com/paper/2406.10721/) (2024) - [Building Cooperative Embodied Agents Modularly with Large Language Models](https://mlsanity.com/paper/2307.02485/) (2023) - [Text2Motion: From Natural Language Instructions to Feasible Plans](https://mlsanity.com/paper/2303.12153/) (2023) - [MolmoAct: Action Reasoning Models that can Reason in Space](https://mlsanity.com/paper/2508.07917/) (2025) - [Text2Reward: Reward Shaping with Language Models for Reinforcement Learning](https://mlsanity.com/paper/2309.11489/) (2023) - [Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration](https://mlsanity.com/paper/2010.09890/) (2020) - [Yell At Your Robot: Improving On-the-Fly from Language Corrections](https://mlsanity.com/paper/2403.12910/) (2024) - [Correcting Robot Plans with Natural Language Feedback](https://mlsanity.com/paper/2204.05186/) (2022) - [Learning Reward Functions by Integrating Human Demonstrations and Preferences](https://mlsanity.com/paper/1906.08928/) (2019) - ["No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomy](https://mlsanity.com/paper/2301.02555/) (2023) - [Learning to Synthesize Programs as Interpretable and Generalizable Policies](https://mlsanity.com/paper/2108.13643/) (2021) - [NL2TL: Transforming Natural Languages to Temporal Logics using Large Language Models](https://mlsanity.com/paper/2305.07766/) (2023) - [The Design of Stretch: A Compact, Lightweight Mobile Manipulator for Indoor Human Environments](https://mlsanity.com/paper/2109.10892/) (2021) - [LATTE: LAnguage Trajectory TransformEr](https://mlsanity.com/paper/2208.02918/) (2022) - [Code as Reward: Empowering Reinforcement Learning with VLMs](https://mlsanity.com/paper/2402.04764/) (2024) - [Data-Efficient Learning of Natural Language to Linear Temporal Logic Translators for Robot Task Specification](https://mlsanity.com/paper/2303.08006/) (2023) - [Learning adaptive planning representations with natural language guidance](https://mlsanity.com/paper/2312.08566/) (2023) - [Learning Trajectory Preferences for Manipulators via Iterative Improvement](https://mlsanity.com/paper/1306.6294/) (2013) - [InterPreT: Interactive Predicate Learning from Language Feedback for Generalizable Task Planning](https://mlsanity.com/paper/2405.19758/) (2024) - [Optimal Target Assignment and Path Finding for Teams of Agents](https://mlsanity.com/paper/1612.05693/) (2016) - [SELP: Generating Safe and Efficient Task Plans for Robot Agents with Large Language Models](https://mlsanity.com/paper/2409.19471/) (2024) - [Prioritized Planning Algorithms for Trajectory Coordination of Multiple Mobile Robots](https://mlsanity.com/paper/1409.2399/) (2014) - [Reshaping Robot Trajectories Using Natural Language Commands: A Study of Multi-Modal Data Alignment Using Transformers](https://mlsanity.com/paper/2203.13411/) (2022) - [Leveraging Environment Interaction for Automated PDDL Translation and Planning with Large Language Models](https://mlsanity.com/paper/2407.12979/) (2024) - [A Survey of Multi-Agent Human-Robot Interaction Systems](https://mlsanity.com/paper/2212.05286/) (2022) - [Towards Collaborative Intelligence: Propagating Intentions and Reasoning for Multi-Agent Coordination with Large Language Models](https://mlsanity.com/paper/2407.12532/) (2024) ## Built on by (most cited first, 3 of 3) - [A Survey on Foundations and Frontiers of Multimodal Agentic Frameworks: Techniques and Applications](https://mlsanity.com/paper/2608.20379/) (2026) - [Multi-Agent Embodied Autonomous Driving (MAEAD): From V2X Information Exchange to Shared World Models](https://mlsanity.com/paper/2606.13840/) (2026) - [Watch, Infer, Coordinate: Inferring Robot Partner Constraints for Zero-Shot Coordination](https://mlsanity.com/paper/2610.02170/) (2026) ## Abstract (arXiv) Successful cooperation among decentralized agents requires each agent to quickly adapt its plan to the behavior of other agents. In scenarios where agents cannot confidently predict one another's intentions and plans, language communication can be crucial for ensuring safety. In this work, we focus on path-level cooperation in which agents must adapt their paths to one another in order to avoid collisions or perform physical collaboration such as joint carrying. In particular, we propose a safe and interpretable multimodal path planning method, CaPE (Code as Path Editor), which generates and updates path plans for an agent based on the environment and language communication from other agents. CaPE leverages a vision-language model (VLM) to synthesize a path editing program verified by a model-based planner, grounding communication to path plan updates in a safe and interpretable way. We evaluate our approach in diverse simulated and real-world scenarios, including multi-robot and human-robot cooperation in autonomous driving, household, and joint carrying tasks. Experimental results demonstrate that CaPE can be integrated into different robotic systems as a plug-and-play module, greatly enhancing a robot's ability to align its plan to language communication from other robots or humans. We also show that the combination of the VLM-based path editing program synthesis and model-based planning safety enables robots to achieve open-ended cooperation while maintaining safety and interpretability.