mlsanityData to 14 Oct 2026

Structural-RNN: Deep Learning on Spatio-Temporal Graphs

arXiv 1511.05298 ↗First posted 17 Nov 2015cs.CV cs.LG cs.NEcited by 234

Not on the map. Why?

Abstract (arXiv)

Deep Recurrent Neural Network architectures, though remarkably capable at modeling sequences, lack an intuitive high-level spatio-temporal structure. That is while many problems in computer vision inherently have an underlying high-level structure and can benefit from it. Spatio-temporal graphs are a popular tool for imposing such high-level intuitions in the formulation of real world problems. In this paper, we propose an approach for combining the power of high-level spatio-temporal graphs and sequence learning success of Recurrent Neural Networks~(RNNs). We develop a scalable method for casting an arbitrary spatio-temporal graph as a rich RNN mixture that is feedforward, fully differentiable, and jointly trainable. The proposed method is generic and principled as it can be used for transforming any spatio-temporal graph through employing a certain set of well defined steps. The evaluations of the proposed approach on a diverse set of problems, ranging from modeling human motion to object interactions, shows improvement over the state-of-the-art with a large margin. We expect this method to empower new approaches to problem formulation through high-level spatio-temporal graphs and Recurrent Neural Networks.

Key facts. Cited by 234 papers in the map (data through 14 Oct 2026). First posted on arXiv 17 Nov 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2019: 1; 2020: 0; 2021: 71; 2022: 48; 2023: 47; 2024: 31; 2025: 22; 2026: 14.

234 papers.

Sort:

By area:

  1. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 485
  2. 2021Conditional Object-Centric Learning from Videocited by 248
  3. 2021Robust Motion In-betweeningcited by 229
  4. 2022Vision GNN: An Image is Worth Graph of Nodescited by 186
  5. 2021GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Camerascited by 153
  6. 2023CG-HOI: Contact-Guided 3D Human-Object Interaction Generationcited by 138
  7. 2021Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoderscited by 109
  8. 2023Avatars Grow Legs: Generating Smooth Human Motion from Sparse Tracking Inputs with Diffusion Modelcited by 107
  9. 20213D AffordanceNet: A Benchmark for Visual Object Affordance Understandingcited by 100
  10. 2022Back to MLP: A Simple Baseline for Human Motion Predictioncited by 100
  11. 2023EqMotion: Equivariant Multi-agent Motion Prediction with Invariant Interaction Reasoningcited by 97
  12. 2021SAGA: Stochastic Whole-Body Grasping with Contactcited by 91
38 more
  1. 2021Space-Time-Separable Graph Convolutional Network for Pose Forecastingcited by 84
  2. 2022BeLFusion: Latent Diffusion for Behavior-Driven Human Motion Predictioncited by 82
  3. 2022ActFormer: A GAN-based Transformer towards General Action-Conditioned 3D Human Motion Generationcited by 77
  4. 2021Multi-Person Extreme Motion Predictioncited by 75
  5. 2021Generating Smooth Pose Sequences for Diverse Human Motion Predictioncited by 66
  6. 2022Progressively Generating Better Initial Guesses Towards Next Stages for High-Quality Human Motion Predictioncited by 66
  7. 2021A Review of Graph Neural Networks and Their Applications in Power Systemscited by 63
  8. 2021Multi-Person 3D Motion Prediction with Multi-Range Transformerscited by 63
  9. 2021Rhythm is a Dancer: Music-Driven Motion Synthesis with Global Structurecited by 53
  10. 2022Intention Aware Robot Crowd Navigation with Attention-Based Interaction Graphcited by 53
  11. 2021TRiPOD: Human Trajectory and Pose Dynamics Forecasting in the Wildcited by 52
  12. 2022Adaptive Trajectory Prediction via Transferable GNNcited by 51
  13. 2023Conformal Decision Theory: Safe Autonomous Decisions from Imperfect Predictionscited by 50
  14. 2021Activity Graph Transformer for Temporal Action Localizationcited by 48
  15. 2022Spatio-Temporal Gating-Adjacency GCN for Human Motion Predictioncited by 48
  16. 2021Forecasting Action through Contact Representations from First Person Videocited by 46
  17. 2021View Vertically: A Hierarchical Network for Trajectory Prediction via Fourier Spectrumscited by 45
  18. 2019Imitation Learning for Human Pose Predictioncited by 44
  19. 2023Hierarchical Generation of Human-Object Interactions with Diffusion Probabilistic Modelscited by 43
  20. 2021Universal Approximation Theorems for Differentiable Geometric Deep Learningcited by 41
  21. 2023Open-Vocabulary Affordance Detection in 3D Point Cloudscited by 41
  22. 2021Unified Graph Structured Models for Video Understandingcited by 38
  23. 2021Multiscale Spatio-Temporal Graph Neural Networks for 3D Skeleton-Based Motion Predictioncited by 37
  24. 2022Interaction Region Visual Transformer for Egocentric Action Anticipationcited by 37
  25. 2021MAAS: Multi-modal Assignation for Active Speaker Detectioncited by 36
  26. 2021RAIN: Reinforced Hybrid Attention Inference Network for Motion Forecastingcited by 36
  27. 2024AMEGO: Active Memory from long EGOcentric videoscited by 36
  28. 2021Spatio-Temporal Graph Dual-Attention Network for Multi-Agent Prediction and Trackingcited by 35
  29. 2021Multi-level Motion Attention for Human Motion Predictioncited by 35
  30. 2023Diverse Human Motion Prediction Guided by Multi-Level Spatial-Temporal Anchorscited by 34
  31. 2023PACE: Human and Camera Motion Estimation from in-the-wild Videoscited by 34
  32. 2022A generic diffusion-based approach for 3D human pose prediction in the wildcited by 33
  33. 2024Deep Learning for Cross-Domain Data Fusion in Urban Computing: Taxonomy, Advances, and Outlookcited by 33
  34. 2021Investigating Pose Representations and Motion Contexts Modeling for 3D Motion Predictioncited by 31
  35. 2022Graph Lifelong Learning: A Surveycited by 30
  36. 20223D Human Motion Prediction: A Surveycited by 30
  37. 2022Motron: Multimodal Probabilistic Human Motion Forecastingcited by 30
  38. 2022Visual Abductive Reasoningcited by 30

50 most cited of 234 All 234 (JSON)