CogDriver: Integrating Cognitive Inertia for Temporally Coherent Planning in Autonomous Driving
On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection
also in 1 more
Abstract (arXiv)
The pursuit of autonomous agents capable of temporally coherent planning is hindered by a fundamental flaw in current vision-language models (VLMs): they lack cognitive inertia. Operating on isolated snapshots, these models cannot form a continuous understanding of the environment, leading to erratic decision jitter and a failure to execute complex, multi-step maneuvers. To remedy this, we introduce CogDriver, a framework designed to build a stable internal representation by instilling this crucial cognitive property. Our work makes two key contributions: (1) We present CogDriver-Data, a large-scale vision-language-action dataset whose narrative annotations provide the supervisory signal for learning temporal dynamics and persistent intent. (2) We develop the CogDriver-Agent, an architecture featuring a sparse temporal memory to maintain a stable internal state. This is enabled by a spatiotemporal knowledge distillation approach that explicitly teaches decision coherence. Comprehensive experiments validate our paradigm: CogDriver-Agent achieves a 22% increase in the closed-loop Driving Score on Bench2Drive and a 21% reduction in mean L2 error on nuScenes, establishing a new state-of-the-art. These significant gains in both long-term decision-making and imitation accuracy provide strong evidence that our agent successfully maintains a temporally coherent internal state, bridging the gap toward more reliable autonomous driving. Project link: https://ocean-luna.github.io/CogDriver.github.io/.
Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 51 references that are in the map. First posted on arXiv 31 Aug 2025.
Builds on
50 of 51 references.
Closest ancestors · 12
Same direction, newest first
- 2025OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning6 Apr
- 2024Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Driving29 Oct
- 2024CarLLaVA: Vision language models for camera-only closed-loop driving14 Jun
- 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models19 Feb
- 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Model16 Feb
- 2024GPT-4V as Traffic Assistant: An In-depth Look at Vision Language Model on Complex Traffic Events3 Feb
- 2023DriveLM: Driving with Graph Visual Question Answering21 Dec
- 2023LingoQA: Visual Question Answering for Autonomous Driving21 Dec
- 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving14 Dec
- 2023Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Driving6 Dec
- 2023Vision Language Models in Autonomous Driving: A Survey and Outlook22 Oct
- 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct
Other references · 38
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2025Qwen2.5-VL Technical Reportcited by 5,587
- 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
- 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
- 2024Qwen2 Technical Reportcited by 2,120
- 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
- 2022Planning-oriented Autonomous Drivingcited by 1,087
- 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 622
- 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
- 2023EVA-02: A Visual Representation for Neon Genesiscited by 381
- 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 368
26 more
- 2023GPT-Driver: Learning to Drive with GPTcited by 347
- 2024VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planningcited by 289
- 2022Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baselinecited by 248
- 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
- 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 232
- 2018Textual Explanations for Self-Driving Vehiclescited by 226
- 2023Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenescited by 191
- 2023Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Drivingcited by 169
- 2023A Language Agent for Autonomous Drivingcited by 164
- 2023MotionLM: Multi-Agent Motion Forecasting as Language Modelingcited by 156
- 2023DriveAdapter: Breaking the Coupling Barrier of Perception and Planning in End-to-End Autonomous Drivingcited by 144
- 2019Talk2Car: Taking Control of Your Self-Driving Carcited by 142
- 2022DRAMA: Joint Risk Localization and Captioning in Drivingcited by 126
- 2024VLP: Vision Language Planning for Autonomous Drivingcited by 126
- 2025DriveTransformer: Unified Transformer for Scalable End-to-End Autonomous Drivingcited by 126
- 2025Don't Shake the Wheel: Momentum-Aware Planning in End-to-End Autonomous Drivingcited by 84
- 2024CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Drivingcited by 77
- 2024A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlookcited by 66
- 2023Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoningcited by 63
- 2023From task structures to world models: What do LLMs know?cited by 46
- 2024DME-Driver: Integrating Human Decision Logic and 3D Scene Perception in Autonomous Drivingcited by 43
- 2024WorldGPT: Empowering LLM as Multimodal World Modelcited by 42
- 2023Open-sourced Data Ecosystem in Autonomous Driving: the Present and Futurecited by 23
- 2025ReAL-AD: Towards Human-Like Reasoning in End-to-End Autonomous Drivingcited by 15
- 2025World Models for Cognitive Agents: Transforming Edge Intelligence in Future Networkscited by 12
- 2025CogAD: Cognitive-Hierarchy Guided End-to-End Autonomous Drivingcited by 4
Built on by
Citing papers per year: 2025: 3; 2026: 2.
5 papers.
- 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 33
- 2026EponaV2: Driving World Model with Comprehensive Future Reasoningcited by 15
- 2025Spatial-aware Vision Language Model for Autonomous Drivingcited by 4
- 2025E3AD: An Emotion-Aware Vision-Language-Action Model for Human-Centric End-to-End Autonomous Drivingcited by 2
- 2026Beyond Textual Chain-of-Thought: A Survey on Action-Grounded Reasoning in Autonomous Drivingcited by 0
- 2026Beyond Textual Chain-of-Thought: A Survey on Action-Grounded Reasoning in Autonomous Driving31 Aug 2026
- 2026EponaV2: Driving World Model with Comprehensive Future Reasoning14 May 2026
- 2025Spatial-aware Vision Language Model for Autonomous Driving30 Dec 2025
- 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future18 Dec 2025
- 2025E3AD: An Emotion-Aware Vision-Language-Action Model for Human-Centric End-to-End Autonomous Driving4 Dec 2025
Similar, not cited
Close in wording, no citation either way
- 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Driving
- 2025ColaVLA: Leveraging Cognitive Latent Reasoning for Hierarchical Parallel Trajectory Planning in Autonomous Driving
- 2026BrainWAM: Action-Space Coordination of Semantic Priors and Predictive Dynamics for Autonomous Driving
- 2026LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Driving
- 2026Cognitive Dual-Process Planning for Autonomous Driving with Structured Scene Knowledge and Verifiable Reasoning-Action Consistency
- 2025Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learning
- 2026SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Driving
- 2025DriveRX: A Vision-Language Reasoning Model for Cross-Task Autonomous Driving