# Brain4Cars: Car That Knows Before You Do via Sensory-Fusion Deep Learning Architecture

arXiv: https://arxiv.org/abs/1601.00740 · HTML: https://mlsanity.com/paper/1601.00740/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jan 2016.

## Built on by (most cited first, 30 of 31)

- [A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlook](https://mlsanity.com/paper/2401.01454/) (2024)
- [AIDE: A Vision-Driven Multi-View, Multi-Modal, Multi-Tasking Dataset for Assistive Driving Perception](https://mlsanity.com/paper/2307.13933/) (2023)
- [StyleDrive: Towards Driving-Style Aware Benchmarking of End-To-End Autonomous Driving](https://mlsanity.com/paper/2506.23982/) (2025)
- [A Comparative Analysis of Decision-Level Fusion for Multimodal Driver Behaviour Understanding](https://mlsanity.com/paper/2204.04734/) (2022)
- [A Survey on Approximate Edge AI for Energy Efficient Autonomous Driving Services](https://mlsanity.com/paper/2304.14271/) (2023)
- [Driver Digital Twin for Online Prediction of Personalized Lane Change Behavior](https://mlsanity.com/paper/2211.01294/) (2022)
- [FlexiFilm: Long Video Generation with Flexible Conditions](https://mlsanity.com/paper/2404.18620/) (2024)
- [METEOR:A Dense, Heterogeneous, and Unstructured Traffic Dataset With Rare Behaviors](https://mlsanity.com/paper/2109.07648/) (2021)
- [A Review of Driver Gaze Estimation and Application in Gaze Behavior Understanding](https://mlsanity.com/paper/2307.01470/) (2023)
- [CEMFormer: Learning to Predict Driver Intentions from In-Cabin and External Cameras via Spatial-Temporal Transformers](https://mlsanity.com/paper/2305.07840/) (2023)
- [Human Action Anticipation: A Survey](https://mlsanity.com/paper/2410.14045/) (2024)
- [PDB: Not All Drivers Are the Same -- A Personalized Dataset for Understanding Driving Behavior](https://mlsanity.com/paper/2503.06477/) (2025)
- [Driver-WM: A Driver-Centric Traffic-Conditioned Latent World Model for In-Cabin Dynamics Rollout](https://mlsanity.com/paper/2605.05092/) (2026)
- [Driving Style Representation in Convolutional Recurrent Neural Network Model of Driver Identification](https://mlsanity.com/paper/2102.05843/) (2021)
- [Artificial Intelligence Methods in In-Cabin Use Cases: A Survey](https://mlsanity.com/paper/2101.02082/) (2021)
- [A Survey on Datasets for Decision-making of Autonomous Vehicle](https://mlsanity.com/paper/2306.16784/) (2023)
- [Looking Inside Out: Anticipating Driver Intent From Videos](https://mlsanity.com/paper/2312.01444/) (2023)
- [Sage Deer: A Super-Aligned Driving Generalist Is Your Copilot](https://mlsanity.com/paper/2505.10257/) (2025)
- [PDB-Eval: An Evaluation of Large Multimodal Models for Description and Explanation of Personalized Driving Behavior](https://mlsanity.com/paper/2507.18447/) (2025)
- [Can Users Specify Driving Speed? Bench2Drive-Speed: Benchmark and Baselines for Desired-Speed Conditioned Autonomous Driving](https://mlsanity.com/paper/2603.25672/) (2026)
- [CaTFormer: Causal Temporal Transformer with Dynamic Contextual Fusion for Driving Intention Prediction](https://mlsanity.com/paper/2507.13425/) (2025)
- [DAOS: A Multimodal In-cabin Behavior Monitoring with Driver Action-Object Synergy Dataset](https://mlsanity.com/paper/2601.11990/) (2026)
- [Driving Like Yourself: A Benchmark for Closed-Loop Personalized End-to-End Autonomous Driving](https://mlsanity.com/paper/2602.18757/) (2026)
- [DriveMotion: A Large-Scale Multi-Source Benchmark for Driver Motion Sequence Modeling and Forecasting](https://mlsanity.com/paper/2609.08117/) (2026)
- [Predicting Driver Intention Using Deep Neural Network](https://mlsanity.com/paper/2105.14790/) (2021)
- [Drivers' Manoeuvre Modelling and Prediction for Safe HRI](https://mlsanity.com/paper/2106.01730/) (2021)
- [Designing deep neural networks for driver intention recognition](https://mlsanity.com/paper/2402.05150/) (2024)
- [Perception Characteristics Distance: Measuring Stability and Robustness of Perception System in Dynamic Conditions under a Certain Decision Rule](https://mlsanity.com/paper/2506.09217/) (2025)
- [V-SenseDrive: A Privacy-Preserving Road Video and In-Vehicle Sensor Fusion Framework for Road Safety & Driver Behaviour Modelling](https://mlsanity.com/paper/2509.18187/) (2025)
- [X-Stream: Exploring MLLMs as Multiplexers for Multi-Stream Understanding](https://mlsanity.com/paper/2606.02482/) (2026)

## Abstract (arXiv)

Advanced Driver Assistance Systems (ADAS) have made driving safer over the last decade. They prepare vehicles for unsafe road conditions and alert drivers if they perform a dangerous maneuver. However, many accidents are unavoidable because by the time drivers are alerted, it is already too late. Anticipating maneuvers beforehand can alert drivers before they perform the maneuver and also give ADAS more time to avoid or prepare for the danger. In this work we propose a vehicular sensor-rich platform and learning algorithms for maneuver anticipation. For this purpose we equip a car with cameras, Global Positioning System (GPS), and a computing device to capture the driving context from both inside and outside of the car. In order to anticipate maneuvers, we propose a sensory-fusion deep learning architecture which jointly learns to anticipate and fuse multiple sensory streams. Our architecture consists of Recurrent Neural Networks (RNNs) that use Long Short-Term Memory (LSTM) units to capture long temporal dependencies. We propose a novel training procedure which allows the network to predict the future given only a partial temporal context. We introduce a diverse data set with 1180 miles of natural freeway and city driving, and show that we can anticipate maneuvers 3.5 seconds before they occur in real-time with a precision and recall of 90.5% and 87.4% respectively.