# Understandable Controller Extraction from Video Observations of Swarms

arXiv: https://arxiv.org/abs/2209.01118 · HTML: https://mlsanity.com/paper/2209.01118/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 31 references that are in the map.
- First posted on arXiv 2 Sep 2022.

## Builds on (most cited first, 5 of 5)

- [Learning Vision-based Flight in Drone Swarms by Imitation](https://mlsanity.com/paper/1908.02999/) (2019)
- [Turing learning: a metric-free approach to inferring behavior and its application to swarms](https://mlsanity.com/paper/1603.04904/) (2016)
- [Jaccard/Tanimoto similarity test and estimation methods](https://mlsanity.com/paper/1903.11372/) (2019)
- [Clone Swarms: Learning to Predict and Control Multi-Robot Systems by Imitation](https://mlsanity.com/paper/1912.02811/) (2019)
- [The Holy Grail of Multi-Robot Planning: Learning to Generate Online-Scalable Solutions from Offline-Optimal Experts](https://mlsanity.com/paper/2107.12254/) (2021)

## Built on by (most cited first, 1 of 1)

- [Show me what you want: Inverse reinforcement learning to automatically design robot swarms by demonstration](https://mlsanity.com/paper/2301.06864/) (2023)

## Abstract (arXiv)

Swarm behavior emerges from the local interaction of agents and their environment often encoded as simple rules. Extracting the rules by watching a video of the overall swarm behavior could help us study and control swarm behavior in nature, or artificial swarms that have been designed by external actors. It could also serve as a new source of inspiration for swarm robotics. Yet extracting such rules is challenging as there is often no visible link between the emergent properties of the swarm and their local interactions. To this end, we develop a method to automatically extract understandable swarm controllers from video demonstrations. The method uses evolutionary algorithms driven by a fitness function that compares eight high-level swarm metrics. The method is able to extract many controllers (behavior trees) in a simple collective movement task. We then provide a qualitative analysis of behaviors that resulted in different trees, but similar behaviors. This provides the first steps toward automatic extraction of swarm controllers based on observations.