# Deep Virtual-to-Real Distillation for Pedestrian Crossing Prediction

arXiv: https://arxiv.org/abs/2211.00856 · HTML: https://mlsanity.com/paper/2211.00856/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 39 references that are in the map.
- First posted on arXiv 2 Nov 2022.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Long-Term On-Board Prediction of People in Traffic Scenes under Uncertainty](https://mlsanity.com/paper/1711.09026/) (2017)
- [Pedestrian Action Anticipation using Contextual Feature Fusion in Stacked RNNs](https://mlsanity.com/paper/2005.06582/) (2020)
- [Predicting Pedestrian Crossing Intention with Feature Fusion and Spatio-Temporal Attention](https://mlsanity.com/paper/2104.05485/) (2021)
- [Knowledge Distillation for Action Anticipation via Label Smoothing](https://mlsanity.com/paper/2004.07711/) (2020)
- [RNN-based Pedestrian Crossing Prediction using Activity and Pose-related Features](https://mlsanity.com/paper/2008.11647/) (2020)
- [Is attention to bounding boxes all you need for pedestrian action prediction?](https://mlsanity.com/paper/2107.08031/) (2021)
- [Coupling Intent and Action for Pedestrian Crossing Behavior Prediction](https://mlsanity.com/paper/2105.04133/) (2021)
- [TrouSPI-Net: Spatio-temporal attention on parallel atrous convolutions and U-GRUs for skeletal pedestrian crossing prediction](https://mlsanity.com/paper/2109.00953/) (2021)

## Built on by (most cited first, 10 of 10)

- [Synthetic Data Generation Framework, Dataset, and Efficient Deep Model for Pedestrian Intention Prediction](https://mlsanity.com/paper/2401.06757/) (2024)
- [Multi-Context Fusion Transformer for Pedestrian Crossing Intention Prediction in Urban Environments](https://mlsanity.com/paper/2511.20011/) (2025)
- [ACIT: Attention-Guided Cross-Modal Interaction Transformer for Pedestrian Crossing Intention Prediction](https://mlsanity.com/paper/2511.20020/) (2025)
- [TrajFusionNet: Pedestrian Crossing Intention Prediction via Fusion of Sequential and Visual Trajectory Representations](https://mlsanity.com/paper/2508.19866/) (2025)
- [Gating Syn-to-Real Knowledge for Pedestrian Crossing Prediction in Safe Driving](https://mlsanity.com/paper/2409.06707/) (2024)
- [Pedestrian Crossing Intention Prediction Using Multimodal Fusion Network](https://mlsanity.com/paper/2511.20008/) (2025)
- [ARCANE-PedSynth: Synthetic Multi-Pedestrian Datasets with Behavioural Crossing Annotations](https://mlsanity.com/paper/2605.24950/) (2026)
- [Adaptive Cross-Modal Fusion with Sparse Attention for Pedestrian Crossing Intention Prediction](https://mlsanity.com/paper/2607.12293/) (2026)
- [Unified Vision-Centric Pedestrian Crossing Action Prediction via Adaptive Patch Projection and Proactive Spatial Rectification](https://mlsanity.com/paper/2609.07420/) (2026)
- [TrajFusionNet+: Transformer-Based Prediction of Pedestrian Crossing Intention via Fusion of Trajectory Representations and Scene Graphs](https://mlsanity.com/paper/2609.10806/) (2026)

## Abstract (arXiv)

Pedestrian crossing is one of the most typical behavior which conflicts with natural driving behavior of vehicles. Consequently, pedestrian crossing prediction is one of the primary task that influences the vehicle planning for safe driving. However, current methods that rely on the practically collected data in real driving scenes cannot depict and cover all kinds of scene condition in real traffic world. To this end, we formulate a deep virtual to real distillation framework by introducing the synthetic data that can be generated conveniently, and borrow the abundant information of pedestrian movement in synthetic videos for the pedestrian crossing prediction in real data with a simple and lightweight implementation. In order to verify this framework, we construct a benchmark with 4667 virtual videos owning about 745k frames (called Virtual-PedCross-4667), and evaluate the proposed method on two challenging datasets collected in real driving situations, i.e., JAAD and PIE datasets. State-of-the-art performance of this framework is demonstrated by exhaustive experiment analysis. The dataset and code can be downloaded from the website urlhttp://www.lotvs.net/codedata/.