# Multi-Modal Hybrid Architecture for Pedestrian Action Prediction

arXiv: https://arxiv.org/abs/2012.00514 · HTML: https://mlsanity.com/paper/2012.00514/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Nov 2020.

## Built on by (most cited first, 23 of 23)

- [Predicting Pedestrian Crossing Intention with Feature Fusion and Spatio-Temporal Attention](https://mlsanity.com/paper/2104.05485/) (2021)
- [Behavioral Intention Prediction in Driving Scenes: A Survey](https://mlsanity.com/paper/2211.00385/) (2022)
- [VisionTrap: Vision-Augmented Trajectory Prediction Guided by Textual Descriptions](https://mlsanity.com/paper/2407.12345/) (2024)
- [From SLAM to Situational Awareness: Challenges and Survey](https://mlsanity.com/paper/2110.00273/) (2021)
- [Coupling Intent and Action for Pedestrian Crossing Behavior Prediction](https://mlsanity.com/paper/2105.04133/) (2021)
- [GPT-4V Takes the Wheel: Promises and Challenges for Pedestrian Behavior Prediction](https://mlsanity.com/paper/2311.14786/) (2023)
- [Foul prediction with estimated poses from soccer broadcast video](https://mlsanity.com/paper/2402.09650/) (2024)
- [Pedestrian Simulation: A Review](https://mlsanity.com/paper/2102.03289/) (2021)
- [Pedestrian Stop and Go Forecasting with Hybrid Feature Fusion](https://mlsanity.com/paper/2203.02489/) (2022)
- [Pedestrian Intention Prediction via Vision-Language Foundation Models](https://mlsanity.com/paper/2507.04141/) (2025)
- [WatchPed: Pedestrian Crossing Intention Prediction Using Embedded Sensors of Smartwatch](https://mlsanity.com/paper/2208.07441/) (2022)
- [Multi-Context Fusion Transformer for Pedestrian Crossing Intention Prediction in Urban Environments](https://mlsanity.com/paper/2511.20011/) (2025)
- [ACIT: Attention-Guided Cross-Modal Interaction Transformer for Pedestrian Crossing Intention Prediction](https://mlsanity.com/paper/2511.20020/) (2025)
- [Diving Deeper Into Pedestrian Behavior Understanding: Intention Estimation, Action Prediction, and Event Risk Assessment](https://mlsanity.com/paper/2407.00446/) (2024)
- [Can Reasons Help Improve Pedestrian Intent Estimation? A Cross-Modal Approach](https://mlsanity.com/paper/2411.13302/) (2024)
- [Seeing Beyond Frames: Zero-Shot Pedestrian Intention Prediction with Raw Temporal Video and Multimodal Cues](https://mlsanity.com/paper/2507.21161/) (2025)
- [Feature Importance in Pedestrian Intention Prediction: A Context-Aware Review](https://mlsanity.com/paper/2409.07645/) (2024)
- [Pedestrian Intention and Trajectory Prediction in Unstructured Traffic Using IDD-PeD](https://mlsanity.com/paper/2506.22111/) (2025)
- [Occlusion-Aware Diffusion Model for Pedestrian Intention Prediction](https://mlsanity.com/paper/2511.00858/) (2025)
- [Pedestrian Crossing Intention Prediction Using Multimodal Fusion Network](https://mlsanity.com/paper/2511.20008/) (2025)
- [Learning Trajectory-Conditioned Relations to Predict Pedestrian Crossing Behavior](https://mlsanity.com/paper/2301.05796/) (2023)
- [Temporal-contextual Event Learning for Pedestrian Crossing Intent Prediction](https://mlsanity.com/paper/2504.06292/) (2025)
- [MTF-Net: Multi-Modal Temporal Feature Fusion Network for Pedestrian Intention Prediction](https://mlsanity.com/paper/2609.20178/) (2026)

## Abstract (arXiv)

Pedestrian behavior prediction is one of the major challenges for intelligent driving systems in urban environments. Pedestrians often exhibit a wide range of behaviors and adequate interpretations of those depend on various sources of information such as pedestrian appearance, states of other road users, the environment layout, etc. To address this problem, we propose a novel multi-modal prediction algorithm that incorporates different sources of information captured from the environment to predict future crossing actions of pedestrians. The proposed model benefits from a hybrid learning architecture consisting of feedforward and recurrent networks for analyzing visual features of the environment and dynamics of the scene. Using the existing 2D pedestrian behavior benchmarks and a newly annotated 3D driving dataset, we show that our proposed model achieves state-of-the-art performance in pedestrian crossing prediction.