# Scalable Perception-Action-Communication Loops with Convolutional and Graph Neural Networks

arXiv: https://arxiv.org/abs/2106.13358 · HTML: https://mlsanity.com/paper/2106.13358/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 49 references that are in the map.
- First posted on arXiv 24 Jun 2021.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [An Algorithmic Perspective on Imitation Learning](https://mlsanity.com/paper/1811.06711/) (2018)
- [Graph Signal Processing: Overview, Challenges and Applications](https://mlsanity.com/paper/1712.00468/) (2017)
- [CAD2RL: Real Single-Image Flight without a Single Real Image](https://mlsanity.com/paper/1611.04201/) (2016)
- [Stability Properties of Graph Neural Networks](https://mlsanity.com/paper/1905.04497/) (2019)
- [Convolutional Neural Network Architectures for Signals Supported on Graphs](https://mlsanity.com/paper/1805.00165/) (2018)
- [Graph Neural Networks: Architectures, Stability and Transferability](https://mlsanity.com/paper/2008.01767/) (2020)
- [Graphs, Convolutions, and Neural Networks: From Graph Filters to Graph Neural Networks](https://mlsanity.com/paper/2003.03777/) (2020)
- [Learning Decentralized Controllers for Robot Swarms with Graph Neural Networks](https://mlsanity.com/paper/1903.10527/) (2019)
- [Gated Graph Recurrent Neural Networks](https://mlsanity.com/paper/2002.01038/) (2020)
- [Delving into Robust Object Detection from Unmanned Aerial Vehicles: A Deep Nuisance Disentanglement Approach](https://mlsanity.com/paper/1908.03856/) (2019)
- [Toward Low-Flying Autonomous MAV Trail Navigation using Deep Neural Networks for Environmental Awareness](https://mlsanity.com/paper/1705.02550/) (2017)
- [Learning Vision-based Flight in Drone Swarms by Imitation](https://mlsanity.com/paper/1908.02999/) (2019)
- [Synthesizing Decentralized Controllers with Graph Neural Networks and Imitation Learning](https://mlsanity.com/paper/2012.14906/) (2020)
- [Distributed Linear-Quadratic Control with Graph Neural Networks](https://mlsanity.com/paper/2103.08417/) (2021)
- [Decentralization of Multiagent Policies by Learning What to Communicate](https://mlsanity.com/paper/1901.08490/) (2019)
- [Discriminability of Single-Layer Graph Neural Networks](https://mlsanity.com/paper/2010.08847/) (2020)
- [Feedback Enhanced Motion Planning for Autonomous Vehicles](https://mlsanity.com/paper/2007.05794/) (2020)
- [Outdoor flocking and formation flight with autonomous aerial robots](https://mlsanity.com/paper/1402.3588/) (2014)
- [Vision-based Multi-MAV Localization with Anonymous Relative Measurements Using Coupled Probabilistic Data Association Filter](https://mlsanity.com/paper/1909.08200/) (2019)

## Built on by (most cited first, 7 of 7)

- [Bag of Tricks for Training Deeper Graph Neural Networks: A Comprehensive Benchmark Study](https://mlsanity.com/paper/2108.10521/) (2021)
- [Symbolic Learning to Optimize: Towards Interpretability and Scalability](https://mlsanity.com/paper/2203.06578/) (2022)
- [Beyond Robustness: A Taxonomy of Approaches towards Resilient Multi-Robot Systems](https://mlsanity.com/paper/2109.12343/) (2021)
- [LPAC: Learnable Perception-Action-Communication Loops with Applications to Coverage Control](https://mlsanity.com/paper/2401.04855/) (2024)
- [Resilient robot teams: a review integrating decentralised control, change-detection, and learning](https://mlsanity.com/paper/2204.10063/) (2022)
- [A Scalable Multi-Robot Framework for Decentralized and Asynchronous Perception-Action-Communication Loops](https://mlsanity.com/paper/2309.10164/) (2023)
- [GenEscape: Hierarchical Multi-Agent Generation of Escape Room Puzzles](https://mlsanity.com/paper/2506.21839/) (2025)

## Abstract (arXiv)

In this paper, we present a perception-action-communication loop design using Vision-based Graph Aggregation and Inference (VGAI). This multi-agent decentralized learning-to-control framework maps raw visual observations to agent actions, aided by local communication among neighboring agents. Our framework is implemented by a cascade of a convolutional and a graph neural network (CNN / GNN), addressing agent-level visual perception and feature learning, as well as swarm-level communication, local information aggregation and agent action inference, respectively. By jointly training the CNN and GNN, image features and communication messages are learned in conjunction to better address the specific task. We use imitation learning to train the VGAI controller in an offline phase, relying on a centralized expert controller. This results in a learned VGAI controller that can be deployed in a distributed manner for online execution. Additionally, the controller exhibits good scaling properties, with training in smaller teams and application in larger teams. Through a multi-agent flocking application, we demonstrate that VGAI yields performance comparable to or better than other decentralized controllers, using only the visual input modality and without accessing precise location or motion state information.