# Visual Action Planning with Multiple Heterogeneous Agents

arXiv: https://arxiv.org/abs/2403.16781 · HTML: https://mlsanity.com/paper/2403.16781/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Mar 2024.

## Built on by (most cited first, 1 of 1)

- [V-MORALS: Visual Morse Graph-Aided Estimation of Regions of Attraction in a Learned Latent Space](https://mlsanity.com/paper/2602.23524/) (2026)

## Abstract (arXiv)

Visual planning methods are promising to handle complex settings where extracting the system state is challenging. However, none of the existing works tackles the case of multiple heterogeneous agents which are characterized by different capabilities and/or embodiment. In this work, we propose a method to realize visual action planning in multi-agent settings by exploiting a roadmap built in a low-dimensional structured latent space and used for planning. To enable multi-agent settings, we infer possible parallel actions from a dataset composed of tuples associated with individual actions. Next, we evaluate feasibility and cost of them based on the capabilities of the multi-agent system and endow the roadmap with this information, building a capability latent space roadmap (C-LSR). Additionally, a capability suggestion strategy is designed to inform the human operator about possible missing capabilities when no paths are found. The approach is validated in a simulated burger cooking task and a real-world box packing task.