# Recurrent Space-time Graph Neural Networks

arXiv: https://arxiv.org/abs/1904.05582 · HTML: https://mlsanity.com/paper/1904.05582/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Apr 2019.

## Built on by (most cited first, 19 of 19)

- [Learnable Fourier Features for Multi-Dimensional Spatial Positional Encoding](https://mlsanity.com/paper/2106.02795/) (2021)
- [Local-Global Context Aware Transformer for Language-Guided Video Segmentation](https://mlsanity.com/paper/2203.09773/) (2022)
- [Dysen-VDM: Empowering Dynamics-aware Text-to-Video Diffusion with LLMs](https://mlsanity.com/paper/2308.13812/) (2023)
- [Enhancing Video-Language Representations with Structural Spatio-Temporal Alignment](https://mlsanity.com/paper/2406.19255/) (2024)
- [Spatio-Temporal Graph Neural Networks: A Survey](https://mlsanity.com/paper/2301.10569/) (2023)
- [Conversation Understanding using Relational Temporal Graph Neural Networks with Auxiliary Cross-Modality Interaction](https://mlsanity.com/paper/2311.04507/) (2023)
- [Dynamic Graph Representation Learning with Neural Networks: A Survey](https://mlsanity.com/paper/2304.05729/) (2023)
- [Space-Time Graph Neural Networks](https://mlsanity.com/paper/2110.02880/) (2021)
- [Intelligent Zero Trust Architecture for 5G/6G Networks: Principles, Challenges, and the Role of Machine Learning in the context of O-RAN](https://mlsanity.com/paper/2105.01478/) (2021)
- [Optimization Planning for 3D ConvNets](https://mlsanity.com/paper/2201.04021/) (2022)
- [Boosting Video Representation Learning with Multi-Faceted Integration](https://mlsanity.com/paper/2201.04023/) (2022)
- [Event Detection on Dynamic Graphs](https://mlsanity.com/paper/2110.12148/) (2021)
- [Representation Power of Graph Neural Networks: Improved Expressivity via Algebraic Analysis](https://mlsanity.com/paper/2205.09801/) (2022)
- [Memory-Augmented Graph Neural Networks: A Brain-Inspired Review](https://mlsanity.com/paper/2209.10818/) (2022)
- [ReFill: Reinforcement Learning for Fill-In Minimization](https://mlsanity.com/paper/2501.16130/) (2025)
- [A3GC-IP: Attention-Oriented Adjacency Adaptive Recurrent Graph Convolutions for Human Pose Estimation from Sparse Inertial Measurements](https://mlsanity.com/paper/2107.11214/) (2021)
- [STAR-GNN: Spatial-Temporal Video Representation for Content-based Retrieval](https://mlsanity.com/paper/2208.06966/) (2022)
- [Collusion Detection with Graph Neural Networks](https://mlsanity.com/paper/2410.07091/) (2024)
- [Push the Limit of Multi-modal Emotion Recognition by Prompting LLMs with Receptive-Field-Aware Attention Weighting](https://mlsanity.com/paper/2411.17674/) (2024)

## Abstract (arXiv)

Learning in the space-time domain remains a very challenging problem in machine learning and computer vision. Current computational models for understanding spatio-temporal visual data are heavily rooted in the classical single-image based paradigm. It is not yet well understood how to integrate information in space and time into a single, general model. We propose a neural graph model, recurrent in space and time, suitable for capturing both the local appearance and the complex higher-level interactions of different entities and objects within the changing world scene. Nodes and edges in our graph have dedicated neural networks for processing information. Nodes operate over features extracted from local parts in space and time and previous memory states. Edges process messages between connected nodes at different locations and spatial scales or between past and present time. Messages are passed iteratively in order to transmit information globally and establish long range interactions. Our model is general and could learn to recognize a variety of high level spatio-temporal concepts and be applied to different learning tasks. We demonstrate, through extensive experiments and ablation studies, that our model outperforms strong baselines and top published methods on recognizing complex activities in video. Moreover, we obtain state-of-the-art performance on the challenging Something-Something human-object interaction dataset.