# GroupNet: Multiscale Hypergraph Neural Networks for Trajectory Prediction with Relational Reasoning

arXiv: https://arxiv.org/abs/2204.08770 · HTML: https://mlsanity.com/paper/2204.08770/

## Key facts

- Cited by 128 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 46 references that are in the map.
- First posted on arXiv 19 Apr 2022.

## Builds on (most cited first, 30 of 31)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networks](https://mlsanity.com/paper/2005.11650/) (2020)
- [VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation](https://mlsanity.com/paper/2005.04259/) (2020)
- [Graph U-Nets](https://mlsanity.com/paper/1905.05178/) (2019)
- [ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst](https://mlsanity.com/paper/1812.03079/) (2018)
- [Neural Relational Inference for Interacting Systems](https://mlsanity.com/paper/1802.04687/) (2018)
- [AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting](https://mlsanity.com/paper/2103.14023/) (2021)
- [Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction](https://mlsanity.com/paper/2002.11927/) (2020)
- [DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents](https://mlsanity.com/paper/1704.04394/) (2017)
- [Spectral Temporal Graph Neural Network for Multivariate Time-series Forecasting](https://mlsanity.com/paper/2103.07719/) (2021)
- [SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints](https://mlsanity.com/paper/1806.01482/) (2018)
- [It Is Not the Journey but the Destination: Endpoint Conditioned Trajectory Prediction](https://mlsanity.com/paper/2004.02025/) (2020)
- [Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks](https://mlsanity.com/paper/1907.03395/) (2019)
- [Convolutional Social Pooling for Vehicle Trajectory Prediction](https://mlsanity.com/paper/1805.06771/) (2018)
- [Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2005.08514/) (2020)
- [Social Attention: Modeling Attention in Human Crowds](https://mlsanity.com/paper/1710.04689/) (2017)
- [Transformer Networks for Trajectory Forecasting](https://mlsanity.com/paper/2003.08111/) (2020)
- [IntentNet: Learning to Predict Intention from Raw Sensor Data](https://mlsanity.com/paper/2101.07907/) (2021)
- [Dynamic Multiscale Graph Neural Networks for 3D Skeleton-Based Human Motion Prediction](https://mlsanity.com/paper/2003.08802/) (2020)
- [Multi-Agent Tensor Fusion for Contextual Trajectory Prediction](https://mlsanity.com/paper/1904.04776/) (2019)
- [Trajectron++: Dynamically-Feasible Trajectory Forecasting With Heterogeneous Data](https://mlsanity.com/paper/2001.03093/) (2020)
- [Collaborative Motion Prediction via Neural Motion Message Passing](https://mlsanity.com/paper/2003.06594/) (2020)
- [Conditional Flow Variational Autoencoders for Structured Sequence Prediction](https://mlsanity.com/paper/1908.09008/) (2019)
- [RAIN: Reinforced Hybrid Attention Inference Network for Motion Forecasting](https://mlsanity.com/paper/2108.01316/) (2021)
- [Online Multi-Agent Forecasting with Interpretable Collaborative Graph Neural Network](https://mlsanity.com/paper/2107.00894/) (2021)
- [Collaborative Uncertainty in Multi-Agent Trajectory Forecasting](https://mlsanity.com/paper/2110.13947/) (2021)

## Built on by (most cited first, 30 of 128)

- [Leapfrog Diffusion Model for Stochastic Trajectory Prediction](https://mlsanity.com/paper/2303.10895/) (2023)
- [EqMotion: Equivariant Multi-agent Motion Prediction with Invariant Interaction Reasoning](https://mlsanity.com/paper/2303.10876/) (2023)
- [Uncovering the Missing Pattern: Unified Framework Towards Trajectory Imputation and Prediction](https://mlsanity.com/paper/2303.16005/) (2023)
- [EigenTrajectory: Low-Rank Descriptors for Multi-Modal Trajectory Forecasting](https://mlsanity.com/paper/2307.09306/) (2023)
- [SingularTrajectory: Universal Trajectory Predictor Using Diffusion Model](https://mlsanity.com/paper/2403.18452/) (2024)
- [FEND: A Future Enhanced Distribution-Aware Contrastive Learning Framework for Long-tail Trajectory Prediction](https://mlsanity.com/paper/2303.16574/) (2023)
- [Can Language Beat Numerical Regression? Language-Based Multimodal Trajectory Prediction](https://mlsanity.com/paper/2403.18447/) (2024)
- [MoFlow: One-Step Flow Matching for Human Trajectory Forecasting via Implicit Maximum Likelihood Estimation based Distillation](https://mlsanity.com/paper/2503.09950/) (2025)
- [TBP-Former: Learning Temporal Bird's-Eye-View Pyramid for Joint Perception and Prediction in Vision-Centric Autonomous Driving](https://mlsanity.com/paper/2303.09998/) (2023)
- [Skeleton-Parted Graph Scattering Networks for 3D Human Motion Prediction](https://mlsanity.com/paper/2208.00368/) (2022)
- [Dynamic-Group-Aware Networks for Multi-Agent Trajectory Prediction with Relational Reasoning](https://mlsanity.com/paper/2206.13114/) (2022)
- [Stochastic Multi-Person 3D Motion Forecasting](https://mlsanity.com/paper/2306.05421/) (2023)
- [SocialCircle: Learning the Angle-based Social Interaction Representation for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2310.05370/) (2023)
- [Trajectory-Aware Body Interaction Transformer for Multi-Person Pose Forecasting](https://mlsanity.com/paper/2303.05095/) (2023)
- [LG-Traj: LLM Guided Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2403.08032/) (2024)
- [R-Pred: Two-Stage Motion Prediction Via Tube-Query Attention-Based Trajectory Refinement](https://mlsanity.com/paper/2211.08609/) (2022)
- [Clifford Group Equivariant Simplicial Message Passing Networks](https://mlsanity.com/paper/2402.10011/) (2024)
- [Adapting to Length Shift: FlexiLength Network for Trajectory Prediction](https://mlsanity.com/paper/2404.00742/) (2024)
- [MART: MultiscAle Relational Transformer Networks for Multi-agent Trajectory Prediction](https://mlsanity.com/paper/2407.21635/) (2024)
- [Ada-MSHyper: Adaptive Multi-Scale Hypergraph Transformer for Time Series Forecasting](https://mlsanity.com/paper/2410.23992/) (2024)
- [Multi-Transmotion: Pre-trained Model for Human Motion Prediction](https://mlsanity.com/paper/2411.02673/) (2024)
- [Auxiliary Tasks Benefit 3D Skeleton-based Human Motion Prediction](https://mlsanity.com/paper/2308.08942/) (2023)
- [Multi-Agent Dynamic Relational Reasoning for Social Robot Navigation](https://mlsanity.com/paper/2401.12275/) (2024)
- [Language-Driven Interactive Traffic Trajectory Generation](https://mlsanity.com/paper/2405.15388/) (2024)
- [Sports-Traj: A Unified Trajectory Generation Model for Multi-Agent Movement in Sports](https://mlsanity.com/paper/2405.17680/) (2024)
- [Reconstructing Groups of People with Hypergraph Relational Reasoning](https://mlsanity.com/paper/2308.15844/) (2023)
- [DICE: Diverse Diffusion Model with Scoring for Trajectory Prediction](https://mlsanity.com/paper/2310.14570/) (2023)
- [Unsupervised Sampling Promoting for Stochastic Human Trajectory Prediction](https://mlsanity.com/paper/2304.04298/) (2023)
- [Simplicial Hopfield networks](https://mlsanity.com/paper/2305.05179/) (2023)
- [Joint-Relation Transformer for Multi-Person Motion Prediction](https://mlsanity.com/paper/2308.04808/) (2023)

## Abstract (arXiv)

Demystifying the interactions among multiple agents from their past trajectories is fundamental to precise and interpretable trajectory prediction. However, previous works only consider pair-wise interactions with limited relational reasoning. To promote more comprehensive interaction modeling for relational reasoning, we propose GroupNet, a multiscale hypergraph neural network, which is novel in terms of both interaction capturing and representation learning. From the aspect of interaction capturing, we propose a trainable multiscale hypergraph to capture both pair-wise and group-wise interactions at multiple group sizes. From the aspect of interaction representation learning, we propose a three-element format that can be learnt end-to-end and explicitly reason some relational factors including the interaction strength and category. We apply GroupNet into both CVAE-based prediction system and previous state-of-the-art prediction systems for predicting socially plausible trajectories with relational reasoning. To validate the ability of relational reasoning, we experiment with synthetic physics simulations to reflect the ability to capture group behaviors, reason interaction strength and interaction category. To validate the effectiveness of prediction, we conduct extensive experiments on three real-world trajectory prediction datasets, including NBA, SDD and ETH-UCY; and we show that with GroupNet, the CVAE-based prediction system outperforms state-of-the-art methods. We also show that adding GroupNet will further improve the performance of previous state-of-the-art prediction systems.