# Predicting Pedestrian Crossing Intention with Feature Fusion and Spatio-Temporal Attention

arXiv: https://arxiv.org/abs/2104.05485 · HTML: https://mlsanity.com/paper/2104.05485/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 36 references that are in the map.
- First posted on arXiv 12 Apr 2021.

## Builds on (most cited first, 19 of 19)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017)
- [Spatiotemporal Relationship Reasoning for Pedestrian Intent Prediction](https://mlsanity.com/paper/2002.08945/) (2020)
- [Pedestrian Action Anticipation using Contextual Feature Fusion in Stacked RNNs](https://mlsanity.com/paper/2005.06582/) (2020)
- [Real-time Intent Prediction of Pedestrians for Autonomous Ground Vehicles via Spatio-Temporal DenseNet](https://mlsanity.com/paper/1904.09862/) (2019)
- [Is the Pedestrian going to Cross? Answering by 2D Pose Estimation](https://mlsanity.com/paper/1807.10580/) (2018)
- [Looking Ahead: Anticipating Pedestrians Crossing with Future Frames Prediction](https://mlsanity.com/paper/1910.09077/) (2019)
- [Multi-Modal Hybrid Architecture for Pedestrian Action Prediction](https://mlsanity.com/paper/2012.00514/) (2020)
- [RNN-based Pedestrian Crossing Prediction using Activity and Pose-related Features](https://mlsanity.com/paper/2008.11647/) (2020)
- [Intention Recognition of Pedestrians and Cyclists by 2D Pose Estimation](https://mlsanity.com/paper/1910.03858/) (2019)
- [FuSSI-Net: Fusion of Spatio-temporal Skeletons for Intention Prediction Network](https://mlsanity.com/paper/2005.07796/) (2020)
- [PePScenes: A Novel Dataset and Baseline for Pedestrian Action Prediction in 3D](https://mlsanity.com/paper/2012.07773/) (2020)

## Built on by (most cited first, 30 of 48)

- [Behavioral Intention Prediction in Driving Scenes: A Survey](https://mlsanity.com/paper/2211.00385/) (2022)
- [A Cognitive-Based Trajectory Prediction Approach for Autonomous Driving](https://mlsanity.com/paper/2402.19251/) (2024)
- [PSI: A Benchmark for Human Interpretation and Response in Traffic Interactions](https://mlsanity.com/paper/2112.02604/) (2021)
- [Multi-modal Fusion Technology based on Vehicle Information: A Survey](https://mlsanity.com/paper/2211.06080/) (2022)
- [Pedestrian Crossing Action Recognition and Trajectory Prediction with 3D Human Keypoints](https://mlsanity.com/paper/2306.01075/) (2023)
- [Assessing Cross-dataset Generalization of Pedestrian Crossing Predictors](https://mlsanity.com/paper/2201.12626/) (2022)
- [GPT-4V Takes the Wheel: Promises and Challenges for Pedestrian Behavior Prediction](https://mlsanity.com/paper/2311.14786/) (2023)
- [PIP-Net: Pedestrian Intention Prediction in the Wild](https://mlsanity.com/paper/2402.12810/) (2024)
- [Deep Virtual-to-Real Distillation for Pedestrian Crossing Prediction](https://mlsanity.com/paper/2211.00856/) (2022)
- [Reasoning-Enhanced Object-Centric Learning for Videos](https://mlsanity.com/paper/2403.15245/) (2024)
- [MambaST: A Plug-and-Play Cross-Spectral Spatial-Temporal Fuser for Efficient Pedestrian Detection](https://mlsanity.com/paper/2408.01037/) (2024)
- [On-Board Pedestrian Trajectory Prediction Using Behavioral Features](https://mlsanity.com/paper/2210.11999/) (2022)
- [Benchmark for Models Predicting Human Behavior in Gap Acceptance Scenarios](https://mlsanity.com/paper/2211.05455/) (2022)
- [Synthetic Data Generation Framework, Dataset, and Efficient Deep Model for Pedestrian Intention Prediction](https://mlsanity.com/paper/2401.06757/) (2024)
- [Vulnerable Road User Detection and Safety Enhancement: A Comprehensive Survey](https://mlsanity.com/paper/2405.19202/) (2024)
- [Surround-view Fisheye BEV-Perception for Valet Parking: Dataset, Baseline and Distortion-insensitive Multi-task Framework](https://mlsanity.com/paper/2212.04111/) (2022)
- [GTransPDM: A Graph-embedded Transformer with Positional Decoupling for Pedestrian Crossing Intention Prediction](https://mlsanity.com/paper/2409.20223/) (2024)
- [WatchPed: Pedestrian Crossing Intention Prediction Using Embedded Sensors of Smartwatch](https://mlsanity.com/paper/2208.07441/) (2022)
- [Local and Global Contextual Features Fusion for Pedestrian Intention Prediction](https://mlsanity.com/paper/2305.01111/) (2023)
- [Context-aware Multi-task Learning for Pedestrian Intent and Trajectory Prediction](https://mlsanity.com/paper/2407.17162/) (2024)
- [Multi-Context Fusion Transformer for Pedestrian Crossing Intention Prediction in Urban Environments](https://mlsanity.com/paper/2511.20011/) (2025)
- [ACIT: Attention-Guided Cross-Modal Interaction Transformer for Pedestrian Crossing Intention Prediction](https://mlsanity.com/paper/2511.20020/) (2025)
- [Analysis over vision-based models for pedestrian action anticipation](https://mlsanity.com/paper/2305.17451/) (2023)
- [Diving Deeper Into Pedestrian Behavior Understanding: Intention Estimation, Action Prediction, and Event Risk Assessment](https://mlsanity.com/paper/2407.00446/) (2024)
- [Application of Vision-Language Model to Pedestrians Behavior and Scene Understanding in Autonomous Driving](https://mlsanity.com/paper/2501.06680/) (2025)
- [MMHU: A Massive-Scale Multimodal Benchmark for Human Behavior Understanding](https://mlsanity.com/paper/2507.12463/) (2025)
- [Seeing Beyond Frames: Zero-Shot Pedestrian Intention Prediction with Raw Temporal Video and Multimodal Cues](https://mlsanity.com/paper/2507.21161/) (2025)
- [TrajFusionNet: Pedestrian Crossing Intention Prediction via Fusion of Sequential and Visual Trajectory Representations](https://mlsanity.com/paper/2508.19866/) (2025)
- [Gating Syn-to-Real Knowledge for Pedestrian Crossing Prediction in Safe Driving](https://mlsanity.com/paper/2409.06707/) (2024)
- [Feature Importance in Pedestrian Intention Prediction: A Context-Aware Review](https://mlsanity.com/paper/2409.07645/) (2024)

## Abstract (arXiv)

Predicting vulnerable road user behavior is an essential prerequisite for deploying Automated Driving Systems (ADS) in the real-world. Pedestrian crossing intention should be recognized in real-time, especially for urban driving. Recent works have shown the potential of using vision-based deep neural network models for this task. However, these models are not robust and certain issues still need to be resolved. First, the global spatio-temproal context that accounts for the interaction between the target pedestrian and the scene has not been properly utilized. Second, the optimum strategy for fusing different sensor data has not been thoroughly investigated. This work addresses the above limitations by introducing a novel neural network architecture to fuse inherently different spatio-temporal features for pedestrian crossing intention prediction. We fuse different phenomena such as sequences of RGB imagery, semantic segmentation masks, and ego-vehicle speed in an optimum way using attention mechanisms and a stack of recurrent neural networks. The optimum architecture was obtained through exhaustive ablation and comparison studies. Extensive comparative experiments on the JAAD pedestrian action prediction benchmark demonstrate the effectiveness of the proposed method, where state-of-the-art performance was achieved. Our code is open-source and publicly available.