# Scene-Aware Conversational ADAS with Generative AI for Real-Time Driver Assistance

arXiv: https://arxiv.org/abs/2507.10500 · HTML: https://mlsanity.com/paper/2507.10500/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 17 references that are in the map.
- First posted on arXiv 14 Jul 2025.

## Builds on (most cited first, 12 of 12)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [A Survey on Multimodal Large Language Models for Autonomous Driving](https://mlsanity.com/paper/2311.12320/) (2023)
- [Vision Language Models in Autonomous Driving: A Survey and Outlook](https://mlsanity.com/paper/2310.14414/) (2023)
- [Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehicles](https://mlsanity.com/paper/2309.10228/) (2023)
- [Personalized Autonomous Driving with Large Language Models: Field Experiments](https://mlsanity.com/paper/2312.09397/) (2023)
- [LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs](https://mlsanity.com/paper/2312.04372/) (2023)
- [Driver Digital Twin for Online Prediction of Personalized Lane Change Behavior](https://mlsanity.com/paper/2211.01294/) (2022)
- [Towards Proactive Interactions for In-Vehicle Conversational Assistants Utilizing Large Language Models](https://mlsanity.com/paper/2403.09135/) (2024)

## Abstract (arXiv)

While autonomous driving technologies continue to advance, current Advanced Driver Assistance Systems (ADAS) remain limited in their ability to interpret scene context or engage with drivers through natural language. These systems typically rely on predefined logic and lack support for dialogue-based interaction, making them inflexible in dynamic environments or when adapting to driver intent. This paper presents Scene-Aware Conversational ADAS (SC-ADAS), a modular framework that integrates Generative AI components including large language models, vision-to-text interpretation, and structured function calling to enable real-time, interpretable, and adaptive driver assistance. SC-ADAS supports multi-turn dialogue grounded in visual and sensor context, allowing natural language recommendations and driver-confirmed ADAS control. Implemented in the CARLA simulator with cloud-based Generative AI, the system executes confirmed user intents as structured ADAS commands without requiring model fine-tuning. We evaluate SC-ADAS across scene-aware, conversational, and revisited multi-turn interactions, highlighting trade-offs such as increased latency from vision-based context retrieval and token growth from accumulated dialogue history. These results demonstrate the feasibility of combining conversational reasoning, scene perception, and modular ADAS control to support the next generation of intelligent driver assistance.