# Enhancing LLM-based Autonomous Driving with Modular Traffic Light and Sign Recognition arXiv: https://arxiv.org/abs/2511.14391 ยท HTML: https://mlsanity.com/paper/2511.14391/ ## Key facts - Cited by 3 papers in the map (data through 14 Oct 2026). - Placed in Chain-of-thought reasoning for driving, one of 17 directions in LLM reasoning. - Builds on 29 of its 39 references that are in the map. - First posted on arXiv 18 Nov 2025. ## Placed in - LLM reasoning / [Chain-of-thought reasoning for driving](https://mlsanity.com/lr/chain-of-thought-reasoning-for-driving/) - End-to-end autonomous driving / [Large language models for driving](https://mlsanity.com/driving/large-language-models-for-driving/) ## Builds on (most cited first, 29 of 29) - [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014) - [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014) - [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015) - [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015) - [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018) - [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017) - [DETRs Beat YOLOs on Real-time Object Detection](https://mlsanity.com/paper/2304.08069/) (2023) - [End-to-end Autonomous Driving: Challenges and Frontiers](https://mlsanity.com/paper/2306.16927/) (2023) - [GPT-Driver: Learning to Drive with GPT](https://mlsanity.com/paper/2310.01415/) (2023) - [LMDrive: Closed-Loop End-to-End Driving with Large Language Models](https://mlsanity.com/paper/2312.07488/) (2023) - [DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models](https://mlsanity.com/paper/2309.16292/) (2023) - [Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation](https://mlsanity.com/paper/2403.17846/) (2024) - [SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignment](https://mlsanity.com/paper/2503.09594/) (2025) - [LLM4Drive: A Survey of Large Language Models for Autonomous Driving](https://mlsanity.com/paper/2311.01043/) (2023) - [Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives](https://mlsanity.com/paper/2501.04003/) (2025) - [OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Driving](https://mlsanity.com/paper/2412.15208/) (2024) - [Language-Grounded Dynamic Scene Graphs for Interactive Object Search with Mobile Manipulation](https://mlsanity.com/paper/2403.08605/) (2024) - [Continuously Learning, Adapting, and Improving: A Dual-Process Approach to Autonomous Driving](https://mlsanity.com/paper/2405.15324/) (2024) - [RT-DETRv3: Real-time End-to-End Object Detection with Hierarchical Dense Positive Supervision](https://mlsanity.com/paper/2409.08475/) (2024) - [BEVDriver: Leveraging BEV Maps in LLMs for Robust Closed-Loop Driving](https://mlsanity.com/paper/2503.03074/) (2025) - [MORE: Mobile Manipulation Rearrangement Through Grounded Language Reasoning](https://mlsanity.com/paper/2505.03035/) (2025) - [Traffic Light Recognition Using Deep Learning and Prior Maps for Autonomous Cars](https://mlsanity.com/paper/1906.11886/) (2019) - [Robotic Task Ambiguity Resolution via Natural Language Interaction](https://mlsanity.com/paper/2504.17748/) (2025) - [YOLO-TS: Real-Time Traffic Sign Detection with Enhanced Accuracy Using Optimized Receptive Fields and Anchor-Free Fusion](https://mlsanity.com/paper/2410.17144/) (2024) - [TLD-READY: Traffic Light Detection -- Relevance Estimation and Deployment Analysis](https://mlsanity.com/paper/2409.07284/) (2024) - [YOLO-LLTS: Real-Time Low-Light Traffic Sign Detection via Prior-Guided Enhancement and Multibranch Feature Interaction](https://mlsanity.com/paper/2503.13883/) (2025) - [The ATLAS of Traffic Lights: A Reliable Perception Framework for Autonomous Driving](https://mlsanity.com/paper/2504.19722/) (2025) - [aUToLights: A Robust Multi-Camera Traffic Light Detection and Tracking System](https://mlsanity.com/paper/2305.08673/) (2023) - [Video-based Traffic Light Recognition by Rockchip RV1126 for Autonomous Driving](https://mlsanity.com/paper/2503.23965/) (2025) ## Built on by (most cited first, 3 of 3) - [Causal Scene Narration with Runtime Safety Supervision for Vision-Language-Action Driving](https://mlsanity.com/paper/2604.01723/) (2026) - [LAD-Drive: Bridging Language and Trajectory with Action-Aware Diffusion Transformers](https://mlsanity.com/paper/2603.02035/) (2026) - [One Perception, All Maneuvers: Directional Traffic Signal Understanding for Maneuver-Level Signal Intent Prediction](https://mlsanity.com/paper/2609.32316/) (2026) ## Abstract (arXiv) Large Language Models (LLMs) are increasingly used for decision-making and planning in autonomous driving, showing promising reasoning capabilities and potential to generalize across diverse traffic situations. However, current LLM-based driving agents lack explicit mechanisms to enforce traffic rules and often struggle to reliably detect small, safety-critical objects such as traffic lights and signs. To address this limitation, we introduce TLS-Assist, a modular redundancy layer that augments LLM-based autonomous driving agents with explicit traffic light and sign recognition. TLS-Assist converts detections into structured natural language messages that are injected into the LLM input, enforcing explicit attention to safety-critical cues. The framework is plug-and-play, model-agnostic, and supports both single-view and multi-view camera setups. We evaluate TLS-Assist in a closed-loop setup on the LangAuto benchmark in CARLA. The results demonstrate relative driving performance improvements of up to 14% over LMDrive and 7% over BEVDriver, while consistently reducing traffic light and sign infractions. We publicly release the code and models on https://github.com/iis-esslingen/TLS-Assist.