# Coupling Intent and Action for Pedestrian Crossing Behavior Prediction

arXiv: https://arxiv.org/abs/2105.04133 · HTML: https://mlsanity.com/paper/2105.04133/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 36 references that are in the map.
- First posted on arXiv 10 May 2021.

## Builds on (most cited first, 19 of 19)

- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioning](https://mlsanity.com/paper/1611.05594/) (2016)
- [Temporal Recurrent Networks for Online Action Detection](https://mlsanity.com/paper/1811.07391/) (2018)
- [RED: Reinforced Encoder-Decoder Networks for Action Anticipation](https://mlsanity.com/paper/1707.04818/) (2017)
- [Unsupervised Traffic Accident Detection in First-Person Videos](https://mlsanity.com/paper/1903.00618/) (2019)
- [Pedestrian Action Anticipation using Contextual Feature Fusion in Stacked RNNs](https://mlsanity.com/paper/2005.06582/) (2020)
- [BiTraP: Bi-directional Pedestrian Trajectory Prediction with Multi-modal Goal Estimation](https://mlsanity.com/paper/2007.14558/) (2020)
- [Egocentric Vision-based Future Vehicle Localization for Intelligent Driving Assistance Systems](https://mlsanity.com/paper/1809.07408/) (2018)
- [Joint Attention in Autonomous Driving (JAAD)](https://mlsanity.com/paper/1609.04741/) (2016)
- [Real-time Intent Prediction of Pedestrians for Autonomous Ground Vehicles via Spatio-Temporal DenseNet](https://mlsanity.com/paper/1904.09862/) (2019)
- [When, Where, and What? A New Dataset for Anomaly Detection in Driving Videos](https://mlsanity.com/paper/2004.03044/) (2020)
- [Is the Pedestrian going to Cross? Answering by 2D Pose Estimation](https://mlsanity.com/paper/1807.10580/) (2018)
- [Multi-Modal Hybrid Architecture for Pedestrian Action Prediction](https://mlsanity.com/paper/2012.00514/) (2020)
- [StartNet: Online Detection of Action Start in Untrimmed Videos](https://mlsanity.com/paper/1903.09868/) (2019)
- [Online Detection of Action Start in Untrimmed, Streaming Videos](https://mlsanity.com/paper/1802.06822/) (2018)
- [Pedestrian Intention Prediction: A Multi-task Perspective](https://mlsanity.com/paper/2010.10270/) (2020)

## Built on by (most cited first, 14 of 14)

- [Behavioral Intention Prediction in Driving Scenes: A Survey](https://mlsanity.com/paper/2211.00385/) (2022)
- [Cognitive Accident Prediction in Driving Scenes: A Multimodality Benchmark](https://mlsanity.com/paper/2212.09381/) (2022)
- [Pedestrian Crossing Action Recognition and Trajectory Prediction with 3D Human Keypoints](https://mlsanity.com/paper/2306.01075/) (2023)
- [Deep Virtual-to-Real Distillation for Pedestrian Crossing Prediction](https://mlsanity.com/paper/2211.00856/) (2022)
- [Pedestrian Stop and Go Forecasting with Hybrid Feature Fusion](https://mlsanity.com/paper/2203.02489/) (2022)
- [On-Board Pedestrian Trajectory Prediction Using Behavioral Features](https://mlsanity.com/paper/2210.11999/) (2022)
- [Diving Deeper Into Pedestrian Behavior Understanding: Intention Estimation, Action Prediction, and Event Risk Assessment](https://mlsanity.com/paper/2407.00446/) (2024)
- [Can Reasons Help Improve Pedestrian Intent Estimation? A Cross-Modal Approach](https://mlsanity.com/paper/2411.13302/) (2024)
- [TrajFusionNet: Pedestrian Crossing Intention Prediction via Fusion of Sequential and Visual Trajectory Representations](https://mlsanity.com/paper/2508.19866/) (2025)
- [Feature Importance in Pedestrian Intention Prediction: A Context-Aware Review](https://mlsanity.com/paper/2409.07645/) (2024)
- [Pedestrian Intention and Trajectory Prediction in Unstructured Traffic Using IDD-PeD](https://mlsanity.com/paper/2506.22111/) (2025)
- [Learning Trajectory-Conditioned Relations to Predict Pedestrian Crossing Behavior](https://mlsanity.com/paper/2301.05796/) (2023)
- [Temporal-contextual Event Learning for Pedestrian Crossing Intent Prediction](https://mlsanity.com/paper/2504.06292/) (2025)
- [From Camera-Based Sensing to Reasoning: A Comprehensive Review Toward Proactive Vulnerable Road User Safety](https://mlsanity.com/paper/2510.03314/) (2025)

## Abstract (arXiv)

Accurate prediction of pedestrian crossing behaviors by autonomous vehicles can significantly improve traffic safety. Existing approaches often model pedestrian behaviors using trajectories or poses but do not offer a deeper semantic interpretation of a person's actions or how actions influence a pedestrian's intention to cross in the future. In this work, we follow the neuroscience and psychological literature to define pedestrian crossing behavior as a combination of an unobserved inner will (a probabilistic representation of binary intent of crossing vs. not crossing) and a set of multi-class actions (e.g., walking, standing, etc.). Intent generates actions, and the future actions in turn reflect the intent. We present a novel multi-task network that predicts future pedestrian actions and uses predicted future action as a prior to detect the present intent and action of the pedestrian. We also designed an attention relation network to incorporate external environmental contexts thus further improve intent and action detection performance. We evaluated our approach on two naturalistic driving datasets, PIE and JAAD, and extensive experiments show significantly improved and more explainable results for both intent detection and action prediction over state-of-the-art approaches. Our code is available at: https://github.com/umautobots/pedestrian_intent_action_detection.