mlsanityData to 7 Oct 2026

ReasonNet: End-to-End Driving with Temporal and Global Reasoning

arXiv 2305.10507 ↗First posted 17 May 2023cs.CV cs.AIcited by 92

On the map: End-to-end autonomous driving → VLMs for autonomous driving

Abstract (arXiv)

The large-scale deployment of autonomous vehicles is yet to come, and one of the major remaining challenges lies in urban dense traffic scenarios. In such cases, it remains challenging to predict the future evolution of the scene and future behaviors of objects, and to deal with rare adverse events such as the sudden appearance of occluded objects. In this paper, we present ReasonNet, a novel end-to-end driving framework that extensively exploits both temporal and global information of the driving scene. By reasoning on the temporal behavior of objects, our method can effectively process the interactions and relationships among features in different frames. Reasoning about the global information of the scene can also improve overall perception performance and benefit the detection of adverse events, especially the anticipation of potential danger from occluded objects. For comprehensive evaluation on occlusion events, we also release publicly a driving simulation benchmark DriveOcclusionSim consisting of diverse occlusion events. We conduct extensive experiments on multiple CARLA benchmarks, where our model outperforms all prior methods, ranking first on the sensor track of the public CARLA Leaderboard.

Key facts. Cited by 92 papers in the map (data through 7 Oct 2026). Placed in VLMs for autonomous driving, one of 9 directions in End-to-end autonomous driving. Builds on 58 of its 66 references that are in the map. First posted on arXiv 17 May 2023.

Builds on

58 of 66 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021NEAT: Neural Attention Fields for End-to-End Autonomous Driving9 Sep

Other references · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2017Graph Attention Networkscited by 8,854
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2016Layer Normalizationcited by 5,579
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  8. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  9. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  10. 2018SlowFast Networks for Video Recognitioncited by 1,862
  11. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,707
  12. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
38 more
  1. 2020Center-based 3D Object Detection and Trackingcited by 1,143
  2. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 844
  3. 2021ByteTrack: Multi-Object Tracking by Associating Every Detection Boxcited by 812
  4. 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
  5. 2020VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representationcited by 528
  6. 2021TrackFormer: Multi-Object Tracking with Transformerscited by 515
  7. 2017End-to-end Driving via Conditional Imitation Learningcited by 479
  8. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 477
  9. 2019PointPainting: Sequential Fusion for 3D Object Detectioncited by 449
  10. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 436
  11. 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Drivingcited by 372
  12. 2020TransTrack: Multiple Object Tracking with Transformercited by 370
  13. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  14. 2019Learning by Cheatingcited by 361
  15. 2022Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Trackingcited by 345
  16. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 343
  17. 2021DenseTNT: End-to-end Trajectory Prediction from Dense Goal Setscited by 292
  18. 2022Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baselinecited by 247
  19. 2021FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Camerascited by 239
  20. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  21. 2022Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformercited by 211
  22. 2021Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentationcited by 200
  23. 2021End-to-End Urban Driving by Imitating a Reinforcement Learning Coachcited by 191
  24. 2021IntentNet: Learning to Predict Intention from Raw Sensor Datacited by 158
  25. 2019End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordancescited by 147
  26. 2020PnPNet: End-to-End Perception and Prediction with Tracking in the Loopcited by 143
  27. 2022Learning from All Vehiclescited by 135
  28. 2020Implicit Latent Variable Model for Scene-Consistent Motion Forecastingcited by 116
  29. 2021Learning to drive from a world on railscited by 114
  30. 2022HDGT: Heterogeneous Driving Graph Transformer for Multi-Agent Trajectory Prediction via Scene Encodingcited by 109
  31. 2020MotionNet: Joint Perception and Motion Prediction for Autonomous Driving Based on Bird's Eye View Mapscited by 100
  32. 2022COOPERNAUT: End-to-End Driving with Cooperative Perception for Networked Vehiclescited by 95
  33. 2019Grounding Human-to-Vehicle Advice for Self-driving Vehiclescited by 71
  34. 2021GRI: General Reinforced Imitation and its Application to Vision-Based Autonomous Drivingcited by 59
  35. 2020End-to-end Contextual Perception and Prediction with Interaction Transformercited by 54
  36. 2020Temporal Interlacing Networkcited by 45
  37. 2022MonoJSG: Joint Semantic and Geometric Cost Volume for Monocular 3D Object Detectioncited by 30
  38. 2020Perceive, Attend, and Drive: Learning Spatial Attention for Safe Self-Drivingcited by 24

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 29; 2025: 34; 2026: 20.

92 papers.

Sort:

By area:

  1. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  2. 2024NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarkingcited by 343
  3. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 301
  4. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuningcited by 252
  5. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  6. 2024Enhancing End-to-End Autonomous Driving with Latent World Modelcited by 161
  7. 2025SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignmentcited by 149
  8. 2025DriveTransformer: Unified Transformer for Scalable End-to-End Autonomous Drivingcited by 125
  9. 2023Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Surveycited by 101
  10. 2023Hidden Biases of End-to-End Driving Modelscited by 95
  11. 2025DriveSuprim: Towards Precise Trajectory Selection for End-to-End Planningcited by 91
  12. 2024VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervisioncited by 85
38 more
  1. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  2. 2024DriveCoT: Integrating Chain-of-Thought Reasoning with End-to-End Drivingcited by 67
  3. 2025Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Modelscited by 60
  4. 2024CarLLaVA: Vision language models for camera-only closed-loop drivingcited by 57
  5. 2024V2X-VLM: End-to-End V2X Cooperative Autonomous Driving Through Large Vision-Language Modelscited by 47
  6. 2023LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programscited by 45
  7. 2025Raw2Drive: Reinforcement Learning with Aligned World Models for End-to-End Autonomous Driving (in CARLA v2)cited by 45
  8. 2024Solving Motion Planning Tasks with a Scalable Generative Modelcited by 40
  9. 2024Can Vehicle Motion Planning Generalize to Realistic Long-tail Scenarios?cited by 38
  10. 2024SmartRefine: A Scenario-Adaptive Refinement Framework for Efficient Motion Predictioncited by 33
  11. 2025BEVDriver: Leveraging BEV Maps in LLMs for Robust Closed-Loop Drivingcited by 26
  12. 2023Open-sourced Data Ecosystem in Autonomous Driving: the Present and Futurecited by 22
  13. 2024VLM-Auto: VLM-based Autonomous Driving Assistant with Human-like Behavior and Understanding for Complex Road Scenescited by 22
  14. 2024Mitigating Covariate Shift in Imitation Learning for Autonomous Vehicles Using Latent Space Generative World Modelscited by 22
  15. 2025ReasonPlan: Unified Scene Prediction and Decision Reasoning for Closed-loop Autonomous Drivingcited by 22
  16. 2025LEAD: Minimizing Learner-Expert Asymmetry in End-to-End Drivingcited by 22
  17. 2024Towards Collaborative Autonomous Driving: Simulation Platform and End-to-End Systemcited by 20
  18. 2026DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Drivingcited by 20
  19. 2023Action-slot: Visual Action-centric Representations for Multi-label Atomic Activity Recognition in Traffic Scenescited by 12
  20. 2023RiskBench: A Scenario-based Benchmark for Risk Identificationcited by 12
  21. 2025Unraveling the Effects of Synthetic Data on End-to-End Autonomous Drivingcited by 12
  22. 2025ResAD: Normalized Residual Trajectory Modeling for End-to-End Autonomous Drivingcited by 12
  23. 2024ComDrive: Comfort-Oriented End-to-End Autonomous Drivingcited by 11
  24. 2025DriveQA: Passing the Driving Knowledge Testcited by 10
  25. 2025AdaDrive: Self-Adaptive Slow-Fast System for Language-Grounded Autonomous Drivingcited by 10
  26. 2025DiffE2E: Rethinking End-to-End Driving with a Hybrid Action Diffusion and Supervised Policycited by 9
  27. 2024CARLA2Real: a tool for reducing the sim2real appearance gap in CARLA simulatorcited by 8
  28. 2025TrajDiff: End-to-end Autonomous Driving without Perception Annotationcited by 8
  29. 2025TakeAD: Preference-based Post-optimization for End-to-end Autonomous Driving with Expert Takeover Datacited by 8
  30. 2024M2DA: Multi-Modal Fusion Transformer Incorporating Driver Attention for Autonomous Drivingcited by 7
  31. 2024SmartPretrain: Model-Agnostic and Dataset-Agnostic Representation Learning for Motion Predictioncited by 7
  32. 2024Pre-trained Transformer-Enabled Strategies with Human-Guided Fine-Tuning for End-to-end Navigation of Autonomous Vehiclescited by 6
  33. 2024DualAD: Dual-Layer Planning for Reasoning in Autonomous Drivingcited by 6
  34. 2025Fully Unified Motion Planning for End-to-End Autonomous Drivingcited by 6
  35. 2025LeAD: The LLM Enhanced Planning System Converged with End-to-end Autonomous Drivingcited by 5
  36. 2025DriveE2E: Closed-Loop Benchmark for End-to-End Autonomous Driving through Real-to-Simulationcited by 5
  37. 2025VR-Drive: Viewpoint-Robust End-to-End Driving with Feed-Forward 3D Gaussian Splattingcited by 5
  38. 2025CoC-VLA: Delving into Adversarial Domain Transfer for Explainable Autonomous Driving via Chain-of-Causality Visual-Language-Action Modelcited by 5

50 most cited of 92 All 92 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Driving
  2. 2023Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Driving
  3. 2026CASCADE: A Spatio-Temporal-Causal Reasoning Representation and Dataset for Driving
  4. 2025AD-EE: Early Exiting for Fast and Reliable Vision-Language Models in Autonomous Driving
  5. 2025World Model-Based End-to-End Scene Generation for Accident Anticipation in Autonomous Driving
  6. 2025From Segments to Scenes: Temporal Understanding for Agentic Autonomous Driving via Vision-Language Models
  7. 2026DriveCombo: Benchmarking Compositional Traffic Rule Reasoning in Autonomous Driving
  8. 2026Reasoning-aware Speculative Decoding for Efficient Vision-Language-Action Models in Autonomous Driving