E3AD: An Emotion-Aware Vision-Language-Action Model for Human-Centric End-to-End Autonomous Driving
On the map: End-to-end autonomous driving → Reasoning VLA and VLM driving policies
Abstract (arXiv)
End-to-end autonomous driving (AD) systems increasingly adopt vision-language-action (VLA) models, yet they typically ignore the passenger's emotional state, which is central to comfort and AD acceptance. We introduce Open-Domain End-to-End (OD-E2E) autonomous driving, where an autonomous vehicle (AV) must interpret free-form natural-language commands, infer the emotion, and plan a physically feasible trajectory. We propose E3AD, an emotion-aware VLA framework that augments semantic understanding with two cognitively inspired components: a continuous Valenc-Arousal-Dominance (VAD) emotion model that captures tone and urgency from language, and a dual-pathway spatial reasoning module that fuses egocentric and allocentric views for human-like spatial cognition. A consistency-oriented training scheme, combining modality pretraining with preference-based alignment, further enforces coherence between emotional intent and driving actions. Across real-world datasets, E3AD improves visual grounding and waypoint planning and achieves state-of-the-art (SOTA) VAD correlation for emotion estimation. These evaluation results show that injecting emotion into VLA-style driving yields more human-aligned grounding, planning, and feedback.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reasoning VLA and VLM driving policies, one of 9 directions in End-to-end autonomous driving. Builds on 59 of its 87 references that are in the map. First posted on arXiv 4 Dec 2025.
Builds on
59 of 87 references.
Closest ancestors · 10
Same direction, newest first
- 2025CogDriver: Integrating Cognitive Inertia for Temporally Coherent Planning in Autonomous Driving31 Aug
- 2025A Survey on Vision-Language-Action Models for Autonomous Driving30 Jun
- 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning16 Jun
- 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Driving9 Jun
- 2025DriveMoE: Mixture-of-Experts for Vision-Language-Action Model in End-to-End Autonomous Driving22 May
- 2025OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Model30 Mar
- 2025SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignment12 Mar
- 2024Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Driving29 Oct
- 2024VLP: Vision Language Planning for Autonomous Driving10 Jan
- 2023GPT-4 Enhanced Multimodal Grounding for Autonomous Driving: Leveraging Cross-Modal Attention with Large Language Models6 Dec
Other references · 49
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2025Qwen3 Technical Reportcited by 8,195
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
- 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
- 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
- 2022Planning-oriented Autonomous Drivingcited by 1,068
- 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
37 more
- 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
- 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
- 2020GoEmotions: A Dataset of Fine-Grained Emotionscited by 479
- 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
- 2021FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detectioncited by 444
- 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Drivingcited by 372
- 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 345
- 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 301
- 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
- 2021TransVG: End-to-End Visual Grounding with Transformerscited by 269
- 2023Universal Instance Perception as Object Discovery and Retrievalcited by 190
- 2023Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Drivingcited by 168
- 2020From Goals, Waypoints & Paths To Long Term Human Trajectory Forecastingcited by 166
- 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 165
- 2019Talk2Car: Taking Control of Your Self-Driving Carcited by 141
- 2025DriveVLA-W0: World Models Amplify Data Scaling Law in Autonomous Drivingcited by 127
- 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspectivecited by 96
- 2024OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Drivingcited by 93
- 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challengescited by 72
- 2022Improving Visual Grounding with Visual-Linguistic Verification and Iterative Reasoningcited by 69
- 2020Goal-GAN: Multimodal Trajectory Prediction Based on Goal Position Estimationcited by 57
- 2024ITINERA: Integrating Spatial Optimization with Large Language Models for Open-domain Urban Itinerary Planningcited by 33
- 2024Sparkle: Mastering Basic Spatial Capabilities in Vision Language Models Elicits Generalization to Spatial Reasoningcited by 33
- 2025CoT-Drive: Efficient Motion Forecasting for Autonomous Driving with LLMs and Chain-of-Thought Promptingcited by 26
- 2023Human-Centric Autonomous Systems With LLMs for User Command Reasoningcited by 22
- 2025Fine-Grained Evaluation of Large Vision-Language Models in Autonomous Drivingcited by 19
- 2020AttnGrounder: Talking to Cars with Attentioncited by 10
- 2020Cosine meets Softmax: A tough-to-beat baseline for visual groundingcited by 10
- 2024A Cognitive-Driven Trajectory Prediction Model for Autonomous Driving in Mixed Autonomy Environmentcited by 10
- 2025Minds on the Move: Decoding Trajectory Prediction in Autonomous Driving with Cognitive Insightscited by 10
- 2025Steerable Adversarial Scenario Generation through Test-Time Preference Alignmentcited by 8
- 2025Mamba-VA: A Mamba-based Approach for Continuous Emotion Recognition in Valence-Arousal Spacecited by 6
- 2022Grounding Commands for Autonomous Vehicles via Layer Fusion with Region-specific Dynamic Layer Attentioncited by 4
- 2021Predicting Physical World Destinations for Commands Given to Self-Driving Carscited by 3
- 2023RouteKG: A knowledge graph-based framework for route prediction on road networkscited by 2
- 2025INTENT: Trajectory Prediction Framework with Intention-Guided Contrastive Clusteringcited by 2
- 2025Think Before You Drive: World Model-Inspired Multimodal Grounding for Autonomous Vehiclescited by 2
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Listen, Look, Drive: Coupling Audio Instructions for User-aware VLA-based Autonomous Driving
- 2026Less Language, More Latents: Annotation-Efficient VLAs for Driving
- 2026UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Driving
- 2026Unifying Language-Action Understanding and Generation for Autonomous Driving
- 2025DriveAction: A Benchmark for Exploring Human-like Driving Decisions in VLA Models
- 2026Vega: Learning to Drive with Natural Language Instructions
- 2025DriveCritic: Towards Context-Aware, Human-Aligned Evaluation for Autonomous Driving with Vision-Language Models
- 2026A Collaborative Multi-Modality Interaction for VLA-based End-to-End Autonomous Driving