# LeapVAD: A Leap in Autonomous Driving via Cognitive Perception and Dual-Process Thinking arXiv: https://arxiv.org/abs/2501.08168 ยท HTML: https://mlsanity.com/paper/2501.08168/ ## Key facts - Cited by 13 papers in the map (data through 14 Oct 2026). - Placed in Chain-of-thought reasoning for driving, one of 17 directions in LLM reasoning. - Builds on 30 of its 71 references that are in the map. - First posted on arXiv 14 Jan 2025. ## Placed in - LLM reasoning / [Chain-of-thought reasoning for driving](https://mlsanity.com/lr/chain-of-thought-reasoning-for-driving/) - End-to-end autonomous driving / [Reasoning VLA and VLM driving policies](https://mlsanity.com/driving/reasoning-vla-and-vlm-driving-policies/) ## Builds on (most cited first, 30 of 30) - [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023) - [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023) - [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023) - [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023) - [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023) - [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023) - [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017) - [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023) - [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023) - [VideoChat: Chat-Centric Video Understanding](https://mlsanity.com/paper/2305.06355/) (2023) - [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://mlsanity.com/paper/2209.06794/) (2022) - [CogVLM: Visual Expert for Pretrained Language Models](https://mlsanity.com/paper/2311.03079/) (2023) - [DriveLM: Driving with Graph Visual Question Answering](https://mlsanity.com/paper/2312.14150/) (2023) - [DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models](https://mlsanity.com/paper/2402.12289/) (2024) - [Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models](https://mlsanity.com/paper/2311.06607/) (2023) - [LMDrive: Closed-Loop End-to-End Driving with Large Language Models](https://mlsanity.com/paper/2312.07488/) (2023) - [MIMIC-IT: Multi-Modal In-Context Instruction Tuning](https://mlsanity.com/paper/2306.05425/) (2023) - [Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?](https://mlsanity.com/paper/2312.03031/) (2023) - [DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving](https://mlsanity.com/paper/2312.09245/) (2023) - [A Language Agent for Autonomous Driving](https://mlsanity.com/paper/2311.10813/) (2023) - [Dolphins: Multimodal Language Model for Driving](https://mlsanity.com/paper/2312.00438/) (2023) - [LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models](https://mlsanity.com/paper/2312.02949/) (2023) - [RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Model](https://mlsanity.com/paper/2402.10828/) (2024) - [DriveCoT: Integrating Chain-of-Thought Reasoning with End-to-End Driving](https://mlsanity.com/paper/2403.16996/) (2024) - [Embodied Understanding of Driving Scenarios](https://mlsanity.com/paper/2403.04593/) (2024) - [Continuously Learning, Adapting, and Improving: A Dual-Process Approach to Autonomous Driving](https://mlsanity.com/paper/2405.15324/) (2024) - [Towards Knowledge-driven Autonomous Driving](https://mlsanity.com/paper/2312.04316/) (2023) - [Towards Artificial General Intelligence (AGI) in the Internet of Things (IoT): Opportunities and Challenges](https://mlsanity.com/paper/2309.07438/) (2023) - [Bringing Diversity to Autonomous Vehicles: An Interpretable Multi-vehicle Decision-making and Planning Framework](https://mlsanity.com/paper/2302.06803/) (2023) - [Building and Testing a General Intelligence Embodied in a Humanoid Robot](https://mlsanity.com/paper/2307.16770/) (2023) ## Built on by (most cited first, 13 of 13) - [A Survey on Vision-Language-Action Models for Autonomous Driving](https://mlsanity.com/paper/2506.24044/) (2025) - [Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future](https://mlsanity.com/paper/2512.16760/) (2025) - [VLM-E2E: Enhancing End-to-End Autonomous Driving with Multimodal Driver Attention Fusion](https://mlsanity.com/paper/2502.18042/) (2025) - [Chain-of-Thought for Autonomous Driving: A Comprehensive Survey and Future Prospects](https://mlsanity.com/paper/2505.20223/) (2025) - [NaviDriveVLM: Decoupling High-Level Reasoning and Motion Planning for Autonomous Driving](https://mlsanity.com/paper/2603.07901/) (2026) - [Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AI](https://mlsanity.com/paper/2510.04978/) (2025) - [Post-Training in End-to-End Autonomous Driving](https://mlsanity.com/paper/2607.08072/) (2026) - [Understanding Adversarial Transferability in Vision-Language Models for Autonomous Driving: A Cross-Architecture Analysis](https://mlsanity.com/paper/2604.27414/) (2026) - [Decision-Making with Lightweight Confidence-Aware Language Model for Autonomous Driving](https://mlsanity.com/paper/2605.25393/) (2026) - [Toward Unified Multimodal Representation Learning for Autonomous Driving](https://mlsanity.com/paper/2603.07874/) (2026) - [Comparative Analysis of Patch Attack on VLM-Based Autonomous Driving Architectures](https://mlsanity.com/paper/2603.08897/) (2026) - [DeepIPCv3: Event-Aware Multi-Modal Sensor Fusion for Sudden Pedestrian Crossing Avoidance](https://mlsanity.com/paper/2606.01277/) (2026) - [LUNA-AD: Lightweight Uncertainty-Aware Language Model with Lifelong Learning for Autonomous Driving](https://mlsanity.com/paper/2606.08470/) (2026) ## Abstract (arXiv) While autonomous driving technology has made remarkable strides, data-driven approaches still struggle with complex scenarios due to their limited reasoning capabilities. Meanwhile, knowledge-driven autonomous driving systems have evolved considerably with the popularization of visual language models. In this paper, we propose LeapVAD, a novel method based on cognitive perception and dual-process thinking. Our approach implements a human-attentional mechanism to identify and focus on critical traffic elements that influence driving decisions. By characterizing these objects through comprehensive attributes - including appearance, motion patterns, and associated risks - LeapVAD achieves more effective environmental representation and streamlines the decision-making process. Furthermore, LeapVAD incorporates an innovative dual-process decision-making module miming the human-driving learning process. The system consists of an Analytic Process (System-II) that accumulates driving experience through logical reasoning and a Heuristic Process (System-I) that refines this knowledge via fine-tuning and few-shot learning. LeapVAD also includes reflective mechanisms and a growing memory bank, enabling it to learn from past mistakes and continuously improve its performance in a closed-loop environment. To enhance efficiency, we develop a scene encoder network that generates compact scene representations for rapid retrieval of relevant driving experiences. Extensive evaluations conducted on two leading autonomous driving simulators, CARLA and DriveArena, demonstrate that LeapVAD achieves superior performance compared to camera-only approaches despite limited training data. Comprehensive ablation studies further emphasize its effectiveness in continuous learning and domain adaptation. Project page: https://pjlab-adg.github.io/LeapVAD/.