# CoLMDriver: LLM-based Negotiation Benefits Cooperative Autonomous Driving

arXiv: https://arxiv.org/abs/2503.08683 · HTML: https://mlsanity.com/paper/2503.08683/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 48 references that are in the map.
- First posted on arXiv 11 Mar 2025.

## Builds on (most cited first, 30 of 41)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [VAD: Vectorized Scene Representation for Efficient Autonomous Driving](https://mlsanity.com/paper/2303.12077/) (2023)
- [DriveLM: Driving with Graph Visual Question Answering](https://mlsanity.com/paper/2312.14150/) (2023)
- [DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models](https://mlsanity.com/paper/2402.12289/) (2024)
- [Multi-Modal Fusion Transformer for End-to-End Autonomous Driving](https://mlsanity.com/paper/2104.09224/) (2021)
- [GPT-Driver: Learning to Drive with GPT](https://mlsanity.com/paper/2310.01415/) (2023)
- [LMDrive: Closed-Loop End-to-End Driving with Large Language Models](https://mlsanity.com/paper/2312.07488/) (2023)
- [Where2comm: Communication-Efficient Collaborative Perception via Spatial Confidence Maps](https://mlsanity.com/paper/2209.12836/) (2022)
- [Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline](https://mlsanity.com/paper/2206.08129/) (2022)
- [DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving](https://mlsanity.com/paper/2312.09245/) (2023)
- [INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps](https://mlsanity.com/paper/1910.03088/) (2019)
- [MP3: A Unified Model to Map, Perceive, Predict and Plan](https://mlsanity.com/paper/2101.06806/) (2021)
- [Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer](https://mlsanity.com/paper/2207.14024/) (2022)
- [DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models](https://mlsanity.com/paper/2309.16292/) (2023)
- [LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving](https://mlsanity.com/paper/2310.03026/) (2023)
- [NEAT: Neural Attention Fields for End-to-End Autonomous Driving](https://mlsanity.com/paper/2109.04456/) (2021)
- [A Survey of Zero-shot Generalisation in Deep Reinforcement Learning](https://mlsanity.com/paper/2111.09794/) (2021)
- [A Language Agent for Autonomous Driving](https://mlsanity.com/paper/2311.10813/) (2023)
- [Learning from All Vehicles](https://mlsanity.com/paper/2203.11934/) (2022)
- [LLM Multi-Agent Systems: Challenges and Open Problems](https://mlsanity.com/paper/2402.03578/) (2024)
- [Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability](https://mlsanity.com/paper/2107.06277/) (2021)
- [MotionNet: Joint Perception and Motion Prediction for Autonomous Driving Based on Bird's Eye View Maps](https://mlsanity.com/paper/2003.06754/) (2020)
- [Collaborative Perception in Autonomous Driving: Methods, Datasets and Challenges](https://mlsanity.com/paper/2301.06262/) (2023)
- [An Extensible Framework for Open Heterogeneous Collaborative Perception](https://mlsanity.com/paper/2401.13964/) (2024)
- [Asynchrony-Robust Collaborative Perception via Bird's Eye View Flow](https://mlsanity.com/paper/2309.16940/) (2023)
- [CarLLaVA: Vision language models for camera-only closed-loop driving](https://mlsanity.com/paper/2406.10165/) (2024)

## Built on by (most cited first, 13 of 13)

- [EponaV2: Driving World Model with Comprehensive Future Reasoning](https://mlsanity.com/paper/2605.14696/) (2026)
- [QuantV2X: A Fully Quantized Multi-Agent System for Cooperative Perception](https://mlsanity.com/paper/2509.03704/) (2025)
- [V2V-GoT: Vehicle-to-Vehicle Cooperative Autonomous Driving with Multimodal Large Language Models and Graph-of-Thoughts](https://mlsanity.com/paper/2509.18053/) (2025)
- [Automated Vehicles Should be Connected with Natural Language](https://mlsanity.com/paper/2507.01059/) (2025)
- [MDrive: Benchmarking Closed-Loop Cooperative Driving for End-to-End Multi-agent Systems](https://mlsanity.com/paper/2605.10904/) (2026)
- [OmniV2X: A Generative Foundation Planner for Efficient End-to-End Cooperative Driving](https://mlsanity.com/paper/2606.21165/) (2026)
- [CMU-Drive and V2V-VLA: Cooperative Multi-agent Unified Driving with Reasoning Benchmark and Vehicle-to-Vehicle Vision-Language-Action Models](https://mlsanity.com/paper/2608.07621/) (2026)
- [CABLE: Cloud-Assisted Bandwidth-efficient LMM-based Encoding for V2X Systems](https://mlsanity.com/paper/2606.19258/) (2026)
- [Vision-Language Assistant for Emotional Reactions to Risky Driving](https://mlsanity.com/paper/2607.16181/) (2026)
- [Multi-Agent Scheduling with LLM-Assisted Contract Net Negotiation for Stream Processing in Mobile Edge Computing](https://mlsanity.com/paper/2608.12371/) (2026)
- [G-MARK: Grounded Multi-Agent Reasoning for Cooperative Driving via Knowledge Graphs](https://mlsanity.com/paper/2608.19964/) (2026)
- [VIPS: Vehicle-Infrastructure Cooperative Planning Benchmark via Pseudo-Simulation](https://mlsanity.com/paper/2609.02462/) (2026)
- [CoLMIN: LLM-based Multi-Decision Path Negotiation for Cooperative Autonomous Driving](https://mlsanity.com/paper/2609.04807/) (2026)

## Abstract (arXiv)

Vehicle-to-vehicle (V2V) cooperative autonomous driving holds great promise for improving safety by addressing the perception and prediction uncertainties inherent in single-agent systems. However, traditional cooperative methods are constrained by rigid collaboration protocols and limited generalization to unseen interactive scenarios. While LLM-based approaches offer generalized reasoning capabilities, their challenges in spatial planning and unstable inference latency hinder their direct application in cooperative driving. To address these limitations, we propose CoLMDriver, the first full-pipeline LLM-based cooperative driving system, enabling effective language-based negotiation and real-time driving control. CoLMDriver features a parallel driving pipeline with two key components: (i) an LLM-based negotiation module under an actor-critic paradigm, which continuously refines cooperation policies through feedback from previous decisions of all vehicles; and (ii) an intention-guided waypoint generator, which translates negotiation outcomes into executable waypoints. Additionally, we introduce InterDrive, a CARLA-based simulation benchmark comprising 10 challenging interactive driving scenarios for evaluating V2V cooperation. Experimental results demonstrate that CoLMDriver significantly outperforms existing approaches, achieving an 11% higher success rate across diverse highly interactive V2V driving scenarios. Code will be released on https://github.com/cxliu0314/CoLMDriver.