# Multimodal Interaction-aware Motion Prediction for Autonomous Street Crossing

arXiv: https://arxiv.org/abs/1808.06887 · HTML: https://mlsanity.com/paper/1808.06887/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Aug 2018.

## Built on by (most cited first, 13 of 13)

- [Bird's-Eye-View Panoptic Segmentation Using Monocular Frontal View Images](https://mlsanity.com/paper/2108.03227/) (2021)
- [3D Multi-Object Tracking Using Graph Neural Networks with Cross-Edge Modality Attention](https://mlsanity.com/paper/2203.10926/) (2022)
- [Transferable and Adaptable Driving Behavior Prediction](https://mlsanity.com/paper/2202.05140/) (2022)
- [CausalAgents: A Robustness Benchmark for Motion Forecasting using Causal Relationships](https://mlsanity.com/paper/2207.03586/) (2022)
- [Is it safe to cross? Interpretable Risk Assessment with GPT-4V for Safety-Aware Street Crossing](https://mlsanity.com/paper/2402.06794/) (2024)
- [Self-Supervised Moving Vehicle Detection from Audio-Visual Cues](https://mlsanity.com/paper/2201.12771/) (2022)
- [Multimodal Sense-Informed Prediction of 3D Human Motions](https://mlsanity.com/paper/2405.02911/) (2024)
- [Evidential Uncertainty Estimation for Multi-Modal Trajectory Prediction](https://mlsanity.com/paper/2503.05274/) (2025)
- [ParkDiffusion: Heterogeneous Multi-Agent Multi-Modal Trajectory Prediction for Automated Parking using Diffusion Models](https://mlsanity.com/paper/2505.00586/) (2025)
- [TrackletMapper: Ground Surface Segmentation and Mapping from Traffic Participant Trajectories](https://mlsanity.com/paper/2209.05247/) (2022)
- [ParkDiffusion++: Ego Intention Conditioned Joint Multi-Agent Trajectory Prediction for Automated Parking using Diffusion Models](https://mlsanity.com/paper/2602.20923/) (2026)
- [Pedestrian Motion State Estimation From 2D Pose](https://mlsanity.com/paper/2103.00145/) (2021)
- [7th AI Driving Olympics: 1st Place Report for Panoptic Tracking](https://mlsanity.com/paper/2112.05210/) (2021)

## Abstract (arXiv)

For mobile robots navigating on sidewalks, it is essential to be able to safely cross street intersections. Most existing approaches rely on the recognition of the traffic light signal to make an informed crossing decision. Although these approaches have been crucial enablers for urban navigation, the capabilities of robots employing such approaches are still limited to navigating only on streets containing signalized intersections. In this paper, we address this challenge and propose a multimodal convolutional neural network framework to predict the safety of a street intersection for crossing. Our architecture consists of two subnetworks; an interaction-aware trajectory estimation stream IA-TCNN, that predicts the future states of all observed traffic participants in the scene, and a traffic light recognition stream AtteNet. Our IA-TCNN utilizes dilated causal convolutions to model the behavior of the observable dynamic agents in the scene without explicitly assigning priorities to the interactions among them. While AtteNet utilizes Squeeze-Excitation blocks to learn a content-aware mechanism for selecting the relevant features from the data, thereby improving the noise robustness. Learned representations from the traffic light recognition stream are fused with the estimated trajectories from the motion prediction stream to learn the crossing decision. Furthermore, we extend our previously introduced Freiburg Street Crossing dataset with sequences captured at different types of intersections, demonstrating complex interactions among the traffic participants. Extensive experimental evaluations on public benchmark datasets and our proposed dataset demonstrate that our network achieves state-of-the-art performance for each of the subtasks, as well as for the crossing safety prediction.