# CEMFormer: Learning to Predict Driver Intentions from In-Cabin and External Cameras via Spatial-Temporal Transformers

arXiv: https://arxiv.org/abs/2305.07840 · HTML: https://mlsanity.com/paper/2305.07840/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 38 references that are in the map.
- First posted on arXiv 13 May 2023.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [MViTv2: Improved Multiscale Vision Transformers for Classification and Detection](https://mlsanity.com/paper/2112.01526/) (2021)
- [DeiT III: Revenge of the ViT](https://mlsanity.com/paper/2204.07118/) (2022)
- [Three things everyone should know about Vision Transformers](https://mlsanity.com/paper/2203.09795/) (2022)
- [Long Short-Term Transformer for Online Action Detection](https://mlsanity.com/paper/2107.03377/) (2021)
- [Recurrent Neural Networks for Driver Activity Anticipation via Sensory-Fusion Architecture](https://mlsanity.com/paper/1509.05016/) (2015)
- [Fine-tuning Image Transformers using Learnable Memory](https://mlsanity.com/paper/2203.15243/) (2022)
- [Brain4Cars: Car That Knows Before You Do via Sensory-Fusion Deep Learning Architecture](https://mlsanity.com/paper/1601.00740/) (2016)
- [An End-to-End Visual-Audio Attention Network for Emotion Recognition in User-Generated Videos](https://mlsanity.com/paper/2003.00832/) (2020)
- [Lattice Long Short-Term Memory for Human Action Recognition](https://mlsanity.com/paper/1708.03958/) (2017)
- [Driver Digital Twin for Online Prediction of Personalized Lane Change Behavior](https://mlsanity.com/paper/2211.01294/) (2022)
- [Driver Intention Anticipation Based on In-Cabin and Driving Scene Monitoring](https://mlsanity.com/paper/2006.11557/) (2020)
- [THMA: Tencent HD Map AI System for Creating HD Map Annotations](https://mlsanity.com/paper/2212.11123/) (2022)
- [Vision-Cloud Data Fusion for ADAS: A Lane Change Prediction Case Study](https://mlsanity.com/paper/2112.04042/) (2021)
- [ViT-DD: Multi-Task Vision Transformer for Semi-Supervised Driver Distraction Detection](https://mlsanity.com/paper/2209.09178/) (2022)
- [Augmented Reality-Based Advanced Driver-Assistance System for Connected Vehicles](https://mlsanity.com/paper/2008.13381/) (2020)
- [ViTASD: Robust Vision Transformer Baselines for Autism Spectrum Disorder Facial Diagnosis](https://mlsanity.com/paper/2210.16943/) (2022)

## Built on by (most cited first, 9 of 9)

- [A Survey on Multimodal Large Language Models for Autonomous Driving](https://mlsanity.com/paper/2311.12320/) (2023)
- [Leveraging Driver Field-of-View for Multimodal Ego-Trajectory Prediction](https://mlsanity.com/paper/2312.08558/) (2023)
- [Interaction-Aware Personalized Vehicle Trajectory Prediction Using Temporal Graph Neural Networks](https://mlsanity.com/paper/2308.07439/) (2023)
- [Evaluation of Infrastructure-based Warning System on Driving Behaviors-A Roundabout Study](https://mlsanity.com/paper/2312.03891/) (2023)
- [CaTFormer: Causal Temporal Transformer with Dynamic Contextual Fusion for Driving Intention Prediction](https://mlsanity.com/paper/2507.13425/) (2025)
- [DriveMotion: A Large-Scale Multi-Source Benchmark for Driver Motion Sequence Modeling and Forecasting](https://mlsanity.com/paper/2609.08117/) (2026)
- [Latent Uncertainty Representations for Video-based Driver Action and Intention Recognition](https://mlsanity.com/paper/2510.05006/) (2025)
- [Towards Safer and Understandable Driver Intention Prediction](https://mlsanity.com/paper/2510.09200/) (2025)
- [TriDrive: Joint Driver, Vehicle, and Road Modeling for Forecasting and Driver Monitoring](https://mlsanity.com/paper/2609.33000/) (2026)

## Abstract (arXiv)

Driver intention prediction seeks to anticipate drivers' actions by analyzing their behaviors with respect to surrounding traffic environments. Existing approaches primarily focus on late-fusion techniques, and neglect the importance of maintaining consistency between predictions and prevailing driving contexts. In this paper, we introduce a new framework called Cross-View Episodic Memory Transformer (CEMFormer), which employs spatio-temporal transformers to learn unified memory representations for an improved driver intention prediction. Specifically, we develop a spatial-temporal encoder to integrate information from both in-cabin and external camera views, along with episodic memory representations to continuously fuse historical data. Furthermore, we propose a novel context-consistency loss that incorporates driving context as an auxiliary supervision signal to improve prediction performance. Comprehensive experiments on the Brain4Cars dataset demonstrate that CEMFormer consistently outperforms existing state-of-the-art methods in driver intention prediction.