# S2RMs: Spatially Structured Recurrent Modules

arXiv: https://arxiv.org/abs/2007.06533 · HTML: https://mlsanity.com/paper/2007.06533/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jul 2020.

## Built on by (most cited first, 10 of 10)

- [Coordination Among Neural Modules Through a Shared Global Workspace](https://mlsanity.com/paper/2103.01197/) (2021)
- [Neural Production Systems: Learning Rule-Governed Visual Dynamics](https://mlsanity.com/paper/2103.01937/) (2021)
- [Fast and Slow Learning of Recurrent Independent Mechanisms](https://mlsanity.com/paper/2105.08710/) (2021)
- [Dynamic Inference with Neural Interpreters](https://mlsanity.com/paper/2110.06399/) (2021)
- [Transformers with Competitive Ensembles of Independent Mechanisms](https://mlsanity.com/paper/2103.00336/) (2021)
- [Spatiotemporal Data Mining: A Survey on Challenges and Open Problems](https://mlsanity.com/paper/2103.17128/) (2021)
- [Test-time Adaptation with Slot-Centric Models](https://mlsanity.com/paper/2203.11194/) (2022)
- [Joint Attention for Multi-Agent Coordination and Social Learning](https://mlsanity.com/paper/2104.07750/) (2021)
- [Neural Attentive Circuits](https://mlsanity.com/paper/2210.08031/) (2022)
- [Boxhead: A Dataset for Learning Hierarchical Representations](https://mlsanity.com/paper/2110.03628/) (2021)

## Abstract (arXiv)

Capturing the structure of a data-generating process by means of appropriate inductive biases can help in learning models that generalize well and are robust to changes in the input distribution. While methods that harness spatial and temporal structures find broad application, recent work has demonstrated the potential of models that leverage sparse and modular structure using an ensemble of sparingly interacting modules. In this work, we take a step towards dynamic models that are capable of simultaneously exploiting both modular and spatiotemporal structures. We accomplish this by abstracting the modeled dynamical system as a collection of autonomous but sparsely interacting sub-systems. The sub-systems interact according to a topology that is learned, but also informed by the spatial structure of the underlying real-world system. This results in a class of models that are well suited for modeling the dynamics of systems that only offer local views into their state, along with corresponding spatial locations of those views. On the tasks of video prediction from cropped frames and multi-agent world modeling from partial observations in the challenging Starcraft2 domain, we find our models to be more robust to the number of available views and better capable of generalization to novel tasks without additional training, even when compared against strong baselines that perform equally well or better on the training distribution.