# STINet: Spatio-Temporal-Interactive Network for Pedestrian Detection and Trajectory Prediction

arXiv: https://arxiv.org/abs/2005.04255 · HTML: https://mlsanity.com/paper/2005.04255/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 May 2020.

## Built on by (most cited first, 22 of 22)

- [TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving](https://mlsanity.com/paper/2205.15997/) (2022)
- [Multi-Modal Fusion Transformer for End-to-End Autonomous Driving](https://mlsanity.com/paper/2104.09224/) (2021)
- [DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets](https://mlsanity.com/paper/2108.09640/) (2021)
- [3D Object Detection for Autonomous Driving: A Comprehensive Survey](https://mlsanity.com/paper/2206.09474/) (2022)
- [4D-Net for Learned Multi-Modal Alignment](https://mlsanity.com/paper/2109.01066/) (2021)
- [PedFormer: Pedestrian Behavior Prediction via Cross-Modal Attention Modulation and Gated Multitask Learning](https://mlsanity.com/paper/2210.07886/) (2022)
- [Deep Multi-Task Learning for Joint Localization, Perception, and Prediction](https://mlsanity.com/paper/2101.06720/) (2021)
- [Learning by Watching](https://mlsanity.com/paper/2106.05966/) (2021)
- [MoDAR: Using Motion Forecasting for 3D Object Detection in Point Cloud Sequences](https://mlsanity.com/paper/2306.03206/) (2023)
- [Ball Trajectory Inference from Multi-Agent Sports Contexts Using Set Transformer and Hierarchical Bi-LSTM](https://mlsanity.com/paper/2306.08206/) (2023)
- [STTracker: Spatio-Temporal Tracker for 3D Single Object Tracking](https://mlsanity.com/paper/2306.17440/) (2023)
- [Objects do not disappear: Video object detection by single-frame object location anticipation](https://mlsanity.com/paper/2308.04770/) (2023)
- [Joint Perception and Prediction for Autonomous Driving: A Survey](https://mlsanity.com/paper/2412.14088/) (2024)
- [Pedestrian Simulation: A Review](https://mlsanity.com/paper/2102.03289/) (2021)
- [SAPI: Surroundings-Aware Vehicle Trajectory Prediction at Intersections](https://mlsanity.com/paper/2306.01812/) (2023)
- [Weakly and Self-Supervised Class-Agnostic Motion Prediction for Autonomous Driving](https://mlsanity.com/paper/2509.13116/) (2025)
- [Real-time Human-Centric Segmentation for Complex Video Scenes](https://mlsanity.com/paper/2108.07199/) (2021)
- [LeTFuser: Light-weight End-to-end Transformer-Based Sensor Fusion for Autonomous Driving with Multi-Task Learning](https://mlsanity.com/paper/2310.13135/) (2023)
- [To New Beginnings: A Survey of Unified Perception in Autonomous Vehicle Software](https://mlsanity.com/paper/2508.20892/) (2025)
- [Predicting decision-making in the future: Human versus Machine](https://mlsanity.com/paper/2110.04465/) (2021)
- [Semantic and Articulated Pedestrian Sensing Onboard a Moving Vehicle](https://mlsanity.com/paper/2309.06313/) (2023)
- [SenseWalk: Agent-Based Semantic Trajectory Simulation Powered by Large Language Models in Zoned Environments](https://mlsanity.com/paper/2607.00989/) (2026)

## Abstract (arXiv)

Detecting pedestrians and predicting future trajectories for them are critical tasks for numerous applications, such as autonomous driving. Previous methods either treat the detection and prediction as separate tasks or simply add a trajectory regression head on top of a detector. In this work, we present a novel end-to-end two-stage network: Spatio-Temporal-Interactive Network (STINet). In addition to 3D geometry modeling of pedestrians, we model the temporal information for each of the pedestrians. To do so, our method predicts both current and past locations in the first stage, so that each pedestrian can be linked across frames and the comprehensive spatio-temporal information can be captured in the second stage. Also, we model the interaction among objects with an interaction graph, to gather the information among the neighboring objects. Comprehensive experiments on the Lyft Dataset and the recently released large-scale Waymo Open Dataset for both object detection and future trajectory prediction validate the effectiveness of the proposed method. For the Waymo Open Dataset, we achieve a bird-eyes-view (BEV) detection AP of 80.73 and trajectory prediction average displacement error (ADE) of 33.67cm for pedestrians, which establish the state-of-the-art for both tasks.