# Graph-SIM: A Graph-based Spatiotemporal Interaction Modelling for Pedestrian Action Prediction

arXiv: https://arxiv.org/abs/2012.02148 · HTML: https://mlsanity.com/paper/2012.02148/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Dec 2020.

## Built on by (most cited first, 12 of 12)

- [Behavioral Intention Prediction in Driving Scenes: A Survey](https://mlsanity.com/paper/2211.00385/) (2022)
- [PedFormer: Pedestrian Behavior Prediction via Cross-Modal Attention Modulation and Gated Multitask Learning](https://mlsanity.com/paper/2210.07886/) (2022)
- [Pedestrian Crossing Action Recognition and Trajectory Prediction with 3D Human Keypoints](https://mlsanity.com/paper/2306.01075/) (2023)
- [GPT-4V Takes the Wheel: Promises and Challenges for Pedestrian Behavior Prediction](https://mlsanity.com/paper/2311.14786/) (2023)
- [DROID: Driver-centric Risk Object Identification](https://mlsanity.com/paper/2106.13201/) (2021)
- [Pedestrian Simulation: A Review](https://mlsanity.com/paper/2102.03289/) (2021)
- [On-Board Pedestrian Trajectory Prediction Using Behavioral Features](https://mlsanity.com/paper/2210.11999/) (2022)
- [Context-aware Multi-task Learning for Pedestrian Intent and Trajectory Prediction](https://mlsanity.com/paper/2407.17162/) (2024)
- [Diving Deeper Into Pedestrian Behavior Understanding: Intention Estimation, Action Prediction, and Event Risk Assessment](https://mlsanity.com/paper/2407.00446/) (2024)
- [Can Reasons Help Improve Pedestrian Intent Estimation? A Cross-Modal Approach](https://mlsanity.com/paper/2411.13302/) (2024)
- [MulCPred: Learning Multi-modal Concepts for Explainable Pedestrian Action Prediction](https://mlsanity.com/paper/2409.09446/) (2024)
- [ESIA: An Energy-Based Spatiotemporal Interaction-Aware Framework for Pedestrian Intention Prediction](https://mlsanity.com/paper/2604.23728/) (2026)

## Abstract (arXiv)

One of the most crucial yet challenging tasks for autonomous vehicles in urban environments is predicting the future behaviour of nearby pedestrians, especially at points of crossing. Predicting behaviour depends on many social and environmental factors, particularly interactions between road users. Capturing such interactions requires a global view of the scene and dynamics of the road users in three-dimensional space. This information, however, is missing from the current pedestrian behaviour benchmark datasets. Motivated by these challenges, we propose 1) a novel graph-based model for predicting pedestrian crossing action. Our method models pedestrians' interactions with nearby road users through clustering and relative importance weighting of interactions using features obtained from the bird's-eye-view. 2) We introduce a new dataset that provides 3D bounding box and pedestrian behavioural annotations for the existing nuScenes dataset. On the new data, our approach achieves state-of-the-art performance by improving on various metrics by more than 15% in comparison to existing methods. The dataset is available at https://github.com/huawei-noah/datasets/PePScenes.