mlsanityData to 7 Oct 2026

ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Driving

arXiv 2506.08052 ↗First posted 9 Jun 2025cs.CV cs.ROcited by 190

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

also in 1 more
Abstract (arXiv)

Recent studies have explored leveraging the world knowledge and cognitive capabilities of Vision-Language Models (VLMs) to address the long-tail problem in end-to-end autonomous driving. However, existing methods typically formulate trajectory planning as a language modeling task, where physical actions are output in the language space, potentially leading to issues such as format-violating outputs, infeasible actions, and slow inference speeds. In this paper, we propose ReCogDrive, a novel Reinforced Cognitive framework for end-to-end autonomous Driving, unifying driving understanding and planning by integrating an autoregressive model with a diffusion planner. First, to instill human driving cognition into the VLM, we introduce a hierarchical data pipeline that mimics the sequential cognitive process of human drivers through three stages: generation, refinement, and quality control. Building on this cognitive foundation, we then address the language-action mismatch by injecting the VLM's learned driving priors into a diffusion planner to efficiently generate continuous and stable trajectories. Furthermore, to enhance driving safety and reduce collisions, we introduce a Diffusion Group Relative Policy Optimization (DiffGRPO) stage, reinforcing the planner for enhanced safety and comfort. Extensive experiments on the NAVSIM and Bench2Drive benchmarks demonstrate that ReCogDrive achieves state-of-the-art performance. Additionally, qualitative results across diverse driving scenarios and DriveBench highlight the model's scene comprehension. All code, model weights, and datasets will be made publicly available to facilitate subsequent research.

Key facts. Cited by 190 papers in the map (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 89 of its 105 references that are in the map. First posted on arXiv 9 Jun 2025.

Builds on

89 of 105 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Driving19 Dec
  2. 2024WiseAD: Knowledge Augmented End-to-End Autonomous Driving with Vision-Language Model13 Dec
  3. 2024Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Driving29 Oct
  4. 2024Is a 3D-Tokenized LLM the Key to Reliable Autonomous Driving?28 May
  5. 2024Automated Evaluation of Large Vision-Language Models on Self-driving Corner Cases16 Apr
  6. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models19 Feb
  7. 2024Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Models2 Jan
  8. 2023DriveLM: Driving with Graph Visual Question Answering21 Dec
  9. 2023LingoQA: Visual Question Answering for Autonomous Driving21 Dec
  10. 2023Dolphins: Multimodal Language Model for Driving1 Dec
  11. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct

Other references · 78

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  6. 2022Scalable Diffusion Models with Transformerscited by 5,931
  7. 2025Qwen2.5-VL Technical Reportcited by 5,418
  8. 2024Qwen2.5 Technical Reportcited by 4,713
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  10. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  11. 2024GPT-4o System Cardcited by 4,467
  12. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
38 more
  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  3. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  6. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  7. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  8. 2020GLU Variants Improve Transformercited by 1,724
  9. 2019Root Mean Square Layer Normalizationcited by 1,425
  10. 2023Mastering Diverse Domains through World Modelscited by 1,193
  11. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  12. 2022Planning-oriented Autonomous Drivingcited by 1,068
  13. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,040
  14. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  15. 2024Phi-4 Technical Reportcited by 675
  16. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 670
  17. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  18. 2024Open-Sora: Democratizing Efficient Video Production for Allcited by 637
  19. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  20. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehiclescited by 485
  21. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 477
  22. 2018An Algorithmic Perspective on Imitation Learningcited by 455
  23. 2019MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Predictioncited by 417
  24. 2025Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Modelscited by 376
  25. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  26. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  27. 2024NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarkingcited by 343
  28. 2024DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Drivingcited by 324
  29. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 301
  30. 2024VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planningcited by 287
  31. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
  32. 2024Diffusion Policy Policy Optimizationcited by 255
  33. 2024Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Drivingcited by 250
  34. 2022Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baselinecited by 247
  35. 2024Hydra-MDP: End-to-end Multimodal Planning with Multi-target Hydra-Distillationcited by 244
  36. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  37. 2023Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenescited by 191
  38. 2022Guided Conditional Diffusion for Controllable Traffic Simulationcited by 189

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 36; 2026: 154.

190 papers.

Sort:

By area:

  1. 2025Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tailcited by 136
  2. 2025DriveVLA-W0: World Models Amplify Data Scaling Law in Autonomous Drivingcited by 127
  3. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  4. 2025AdaThinkDrive: Adaptive Thinking via Reinforcement Learning for Autonomous Drivingcited by 57
  5. 2025DiffusionDriveV2: Reinforcement Learning-Constrained Truncated Diffusion Modeling in End-to-End Autonomous Drivingcited by 52
  6. 2025DriveLaW:Unifying Planning and Video Generation in a Latent Driving Worldcited by 43
  7. 2026Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot Embodimentscited by 42
  8. 2025DriveDPO: Policy Learning via Safety DPO For End-to-End Autonomous Drivingcited by 39
  9. 2025Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learningcited by 38
  10. 2026Driving on Registerscited by 38
  11. 2026DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generationcited by 36
  12. 2025SimScale: Learning to Drive via Real-World Simulation at Scalecited by 34
38 more
  1. 2026SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Drivingcited by 34
  2. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  3. 2025IRL-VLA: Training an Vision-Language-Action Policy via Reward World Modelcited by 31
  4. 2026DriveWorld-VLA: Unified Latent-Space World Modeling with Vision-Language-Action for Autonomous Drivingcited by 31
  5. 2026Latent-WAM: Latent World Action Modeling for End-to-End Autonomous Drivingcited by 27
  6. 2026LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Drivingcited by 26
  7. 2025MindDrive: A Vision-Language-Action Model for Autonomous Driving via Online Reinforcement Learningcited by 25
  8. 2026VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Drivingcited by 19
  9. 2026DriveVA: Video Action Models are Zero-Shot Driverscited by 19
  10. 2026DriveFine: Refining-Augmented Masked Diffusion VLA for Precise and Robust Drivingcited by 18
  11. 2026Unleashing VLA Potentials in Autonomous Driving via Explicit Learning from Failurescited by 17
  12. 2026DriveDreamer-Policy: A Geometry-Grounded World-Action Model for Unified Generation and Planningcited by 17
  13. 2026UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Drivingcited by 17
  14. 2025Zero-Human Demonstration End-to-end Autonomous Driving with Trajectory Scorercited by 16
  15. 2025WAM-Flow: Parallel Coarse-to-Fine Motion Planning via Discrete Flow Matching for Autonomous Drivingcited by 15
  16. 2025DrivePI: Spatial-aware 4D MLLM for Unified Autonomous Driving Understanding, Perception, Prediction and Planningcited by 15
  17. 2026NoRD: A Data-Efficient Vision-Language-Action Model that Drives without Reasoningcited by 15
  18. 2026SpanVLA: Efficient Action Bridging and Learning from Negative-Recovery Samples for Vision-Language-Action Modelcited by 15
  19. 2026DriveFuture: Future-Aware Latent World Models for Autonomous Drivingcited by 15
  20. 2025Rethinking Driving World Model as Synthetic Data Generator for Perception Taskscited by 14
  21. 2025dVLM-AD: Enhance Diffusion Vision-Language-Model for Driving via Controllable Reasoningcited by 14
  22. 2026Thinking with Geometry: Active Geometry Integration for Spatial Reasoningcited by 14
  23. 2025AD-R1: Closed-Loop Reinforcement Learning for End-to-End Autonomous Driving with Impartial World Modelscited by 13
  24. 2026DVGT-2: Vision-Geometry-Action Model for Autonomous Driving at Scalecited by 12
  25. 2026EponaV2: Driving World Model with Comprehensive Future Reasoningcited by 12
  26. 2026DriveWAM: Video Generative Priors Enable Scalable World-Action Modeling for Autonomous Drivingcited by 12
  27. 2025OpenREAD: Reinforced Open-Ended Reasoning for End-to-End Autonomous Driving with LLM-as-Criticcited by 11
  28. 2026Unleashing the Potential of Diffusion Models for End-to-End Autonomous Drivingcited by 11
  29. 2026DynVLA: Learning World Dynamics for Action Reasoning in Autonomous Drivingcited by 11
  30. 2025UniUGP: Unifying Understanding, Generation, and Planing For End-to-end Autonomous Drivingcited by 10
  31. 2025WAM-Diff: A Masked Diffusion VLA Framework with MoE and Online Reinforcement Learning for Autonomous Drivingcited by 10
  32. 2025DiffusionVL: Translating Any Autoregressive Models into Diffusion Vision Language Modelscited by 10
  33. 2026ExploreVLA: Dense World Modeling and Exploration for End-to-End Autonomous Drivingcited by 10
  34. 2026CoWorld-VLA: Thinking in a Multi-Expert World Model for Autonomous Drivingcited by 10
  35. 2026Metis: A Generalizable and Efficient World-Action Model for Autonomous Driving and Urban Navigationcited by 10
  36. 2026Qwen-RobotNav Technical Report: A Scalable Navigation Model Designed for an Agentic Navigation Systemcited by 10
  37. 2025MTRDrive: Memory-Tool Synergistic Reasoning for Robust Autonomous Driving in Corner Casescited by 9
  38. 2025InfiniteVL: Synergizing Linear and Sparse Attention for Highly-Efficient, Unlimited-Input Vision-Language Modelscited by 9

50 most cited of 190 All 190 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning
  2. 2025ColaVLA: Leveraging Cognitive Latent Reasoning for Hierarchical Parallel Trajectory Planning in Autonomous Driving
  3. 2025ReAL-AD: Towards Human-Like Reasoning in End-to-End Autonomous Driving
  4. 2025AutoDrive-R2: Incentivizing Reasoning and Self-Reflection Capacity for VLA Model in Autonomous Driving
  5. 2025Discrete Diffusion for Reflective Vision-Language-Action Models in Autonomous Driving
  6. 2025DriveMind: A Dual Visual Language Model-based Reinforcement Learning Framework for Autonomous Driving
  7. 2025InsightDrive: Insight Scene Representation for End-to-End Autonomous Driving
  8. 2025DriveRX: A Vision-Language Reasoning Model for Cross-Task Autonomous Driving