mlsanityData to 7 Oct 2026

Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Driving

arXiv 2305.06242 ↗First posted 10 May 2023cs.CVcited by 168

On the map: End-to-end autonomous driving → End-to-end system architecture

Abstract (arXiv)

End-to-end autonomous driving has made impressive progress in recent years. Existing methods usually adopt the decoupled encoder-decoder paradigm, where the encoder extracts hidden features from raw sensor data, and the decoder outputs the ego-vehicle's future trajectories or actions. Under such a paradigm, the encoder does not have access to the intended behavior of the ego agent, leaving the burden of finding out safety-critical regions from the massive receptive field and inferring about future situations to the decoder. Even worse, the decoder is usually composed of several simple multi-layer perceptrons (MLP) or GRUs while the encoder is delicately designed (e.g., a combination of heavy ResNets or Transformer). Such an imbalanced resource-task division hampers the learning process. In this work, we aim to alleviate the aforementioned problem by two principles: (1) fully utilizing the capacity of the encoder; (2) increasing the capacity of the decoder. Concretely, we first predict a coarse-grained future position and action based on the encoder features. Then, conditioned on the position and action, the future scene is imagined to check the ramification if we drive accordingly. We also retrieve the encoder features around the predicted coordinate to obtain fine-grained information about the safety-critical region. Finally, based on the predicted future and the retrieved salient feature, we refine the coarse-grained position and action by predicting its offset from ground-truth. The above refinement module could be stacked in a cascaded fashion, which extends the capacity of the decoder with spatial-temporal prior knowledge about the conditioned future. We conduct experiments on the CARLA simulator and achieve state-of-the-art performance in closed-loop benchmarks. Extensive ablation studies demonstrate the effectiveness of each proposed module.

Key facts. Cited by 168 papers in the map (data through 7 Oct 2026). Placed in End-to-end system architecture, one of 9 directions in End-to-end autonomous driving. Builds on 77 of its 98 references that are in the map. First posted on arXiv 10 May 2023.

Builds on

77 of 98 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning15 Jul
  2. 2022MMFN: Multi-Modal-Fusion-Net for End-to-End Driving1 Jul
  3. 2022Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline16 Jun
  4. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving31 May
  5. 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Driving19 Apr
  6. 2021End-to-end Interpretable Neural Motion Planner17 Jan

Other references · 71

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2022A ConvNet for the 2020scited by 3,545
  8. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  9. 2015Fast R-CNNcited by 2,930
  10. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  11. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  12. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
38 more
  1. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
  2. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  3. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  4. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  5. 2020Center-based 3D Object Detection and Trackingcited by 1,143
  6. 2022Planning-oriented Autonomous Drivingcited by 1,068
  7. 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,054
  8. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,040
  9. 2016End to End Learning for Self-Driving Carscited by 1,000
  10. 2017PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volumecited by 966
  11. 2018PointRCNN: 3D Object Proposal Generation and Detection from Point Cloudcited by 939
  12. 2019PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detectioncited by 890
  13. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 844
  14. 2018Path Aggregation Network for Instance Segmentationcited by 796
  15. 2016FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networkscited by 780
  16. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  17. 2021BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-Viewcited by 711
  18. 2022BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detectioncited by 568
  19. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 550
  20. 2017End-to-end Driving via Conditional Imitation Learningcited by 479
  21. 2020Voxel R-CNN: Towards High Performance Voxel-based 3D Object Detectioncited by 409
  22. 2018ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worstcited by 391
  23. 2019Learning by Cheatingcited by 361
  24. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 343
  25. 2021Categorical Depth Distribution Network for Monocular 3D Object Detectioncited by 326
  26. 2019STD: Sparse-to-Dense 3D Object Detector for Point Cloudcited by 282
  27. 2022Motion Transformer with Global Intention Localization and Local Movement Refinementcited by 267
  28. 2022Cross-view Transformers for real-time Map-view Semantic Segmentationcited by 264
  29. 2021MP3: A Unified Model to Map, Perceive, Predict and Plancited by 211
  30. 2022Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformercited by 211
  31. 2018Orthographic Feature Transform for Monocular 3D Object Detectioncited by 201
  32. 2021End-to-End Urban Driving by Imitating a Reinforcement Learning Coachcited by 191
  33. 2021NEAT: Neural Attention Fields for End-to-End Autonomous Drivingcited by 174
  34. 2020From Goals, Waypoints & Paths To Long Term Human Trajectory Forecastingcited by 166
  35. 2022M2BEV: Multi-Camera Joint 3D Detection and Segmentation with Unified Birds-Eye View Representationcited by 155
  36. 2020Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representationscited by 150
  37. 2022Model-Based Imitation Learning for Urban Drivingcited by 149
  38. 2019End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordancescited by 147

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 15; 2024: 35; 2025: 69; 2026: 49.

168 papers.

Sort:

By area:

  1. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  2. 2024NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarkingcited by 343
  3. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  4. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 301
  5. 2024VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planningcited by 287
  6. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuningcited by 252
  7. 2024Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Drivingcited by 250
  8. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  9. 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Drivingcited by 190
  10. 2025ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generationcited by 177
  11. 2024Enhancing End-to-End Autonomous Driving with Latent World Modelcited by 161
  12. 2025SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignmentcited by 149
38 more
  1. 2023DriveAdapter: Breaking the Coupling Barrier of Perception and Planning in End-to-End Autonomous Drivingcited by 142
  2. 2025OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoningcited by 142
  3. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  4. 2025DriveTransformer: Unified Transformer for Scalable End-to-End Autonomous Drivingcited by 125
  5. 2023Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Surveycited by 101
  6. 2023Visual Point Cloud Forecasting enables Scalable Autonomous Drivingcited by 97
  7. 2023Hidden Biases of End-to-End Driving Modelscited by 95
  8. 2025DriveSuprim: Towards Precise Trajectory Selection for End-to-End Planningcited by 91
  9. 2024VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervisioncited by 85
  10. 2025DriveMoE: Mixture-of-Experts for Vision-Language-Action Model in End-to-End Autonomous Drivingcited by 85
  11. 2024PLUTO: Pushing the Limit of Imitation Learning-based Planning for Autonomous Drivingcited by 84
  12. 2024OccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Drivingcited by 76
  13. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  14. 2024DriveCoT: Integrating Chain-of-Thought Reasoning with End-to-End Drivingcited by 67
  15. 2024CarLLaVA: Vision language models for camera-only closed-loop drivingcited by 57
  16. 2024Think2Drive: Efficient Reinforcement Learning by Thinking in Latent World Model for Quasi-Realistic Autonomous Driving (in CARLA-v2)cited by 51
  17. 2024SparseAD: Sparse Query-Centric Paradigm for Efficient End-to-End Autonomous Drivingcited by 46
  18. 2024Navigation-Guided Sparse Scene Representation for End-to-End Autonomous Drivingcited by 46
  19. 2025Raw2Drive: Reinforcement Learning with Aligned World Models for End-to-End Autonomous Driving (in CARLA v2)cited by 45
  20. 2024End-to-End Autonomous Driving through V2X Cooperationcited by 42
  21. 2025ReSim: Reliable World Simulation for Autonomous Drivingcited by 42
  22. 2025Centaur: Robust End-to-End Autonomous Driving with Test-Time Trainingcited by 41
  23. 2025iPad: Iterative Proposal-centric End-to-End Autonomous Drivingcited by 41
  24. 2024Continuously Learning, Adapting, and Improving: A Dual-Process Approach to Autonomous Drivingcited by 40
  25. 2025Hydra-NeXt: Robust Closed-Loop Driving with Open-Loop Trainingcited by 39
  26. 2025HiP-AD: Hierarchical and Multi-Granularity Planning with Deformable Attention for Autonomous Driving in a Single Decodercited by 35
  27. 2025CaRL: Learning Scalable Planning Policies with Simple Rewardscited by 34
  28. 2025SimScale: Learning to Drive via Real-World Simulation at Scalecited by 34
  29. 2026SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Drivingcited by 34
  30. 2023PPAD: Iterative Interactions of Prediction and Planning for End-to-end Autonomous Drivingcited by 33
  31. 2023MUVO: A Multimodal Generative World Model for Autonomous Driving with Geometric Representationscited by 33
  32. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  33. 2024Hidden Biases of End-to-End Driving Datasetscited by 30
  34. 2025SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Drivingcited by 28
  35. 2025DiffAD: A Unified Diffusion Modeling Approach for Autonomous Drivingcited by 26
  36. 2023Rethinking Imitation-based Planner for Autonomous Drivingcited by 25
  37. 2024GraphAD: Interaction Scene Graph for End-to-end Autonomous Drivingcited by 25
  38. 2025MindDrive: A Vision-Language-Action Model for Autonomous Driving via Online Reinforcement Learningcited by 25

50 most cited of 168 All 168 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023End-to-end Autonomous Driving using Deep Learning: A Systematic Review
  2. 2025Decoupling Scene Perception and Ego Status: A Multi-Context Fusion Approach for Enhanced Generalization in End-to-End Autonomous Driving
  3. 2021Increasing the Efficiency of Policy Learning for Autonomous Vehicles by Multi-Task Representation Learning
  4. 2024DualAD: Disentangling the Dynamic and Static World for End-to-End Driving
  5. 2026DriveMamba: Task-Centric Scalable State Space Model for Efficient End-to-End Autonomous Driving
  6. 2026A Survey on End-to-End Autonomous Driving Training from the Perspectives of Data, Strategy, and Platform
  7. 2026End-to-End Learning vs. Modular Architectures: Comparative Insights into Autonomous Driving Systems
  8. 2023ADriver-I: A General World Model for Autonomous Driving