mlsanityData to 7 Oct 2026

ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning

arXiv 2207.07601 ↗First posted 15 Jul 2022cs.CVcited by 365

On the map: End-to-end autonomous driving → End-to-end system architecture

Abstract (arXiv)

Many existing autonomous driving paradigms involve a multi-stage discrete pipeline of tasks. To better predict the control signals and enhance user safety, an end-to-end approach that benefits from joint spatial-temporal feature learning is desirable. While there are some pioneering works on LiDAR-based input or implicit design, in this paper we formulate the problem in an interpretable vision-based setting. In particular, we propose a spatial-temporal feature learning scheme towards a set of more representative features for perception, prediction and planning tasks simultaneously, which is called ST-P3. Specifically, an egocentric-aligned accumulation technique is proposed to preserve geometry information in 3D space before the bird's eye view transformation for perception; a dual pathway modeling is devised to take past motion variations into account for future prediction; a temporal-based refinement unit is introduced to compensate for recognizing vision-based elements for planning. To the best of our knowledge, we are the first to systematically investigate each part of an interpretable end-to-end vision-based autonomous driving system. We benchmark our approach against previous state-of-the-arts on both open-loop nuScenes dataset as well as closed-loop CARLA simulation. The results show the effectiveness of our method. Source code, model and protocol details are made publicly available at https://github.com/OpenPerceptionX/ST-P3.

Key facts. Cited by 365 papers in the map (data through 7 Oct 2026). Placed in End-to-end system architecture, one of 9 directions in End-to-end autonomous driving. Builds on 46 of its 56 references that are in the map. First posted on arXiv 15 Jul 2022.

Builds on

46 of 56 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Driving19 Apr
  2. 2021Learning Interpretable End-to-End Vision-Based Motion Planning for Autonomous Driving with Optical Flow Distillation18 Apr
  3. 2021End-to-end Interpretable Neural Motion Planner17 Jan

Other references · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  3. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  4. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  5. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  6. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  7. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,040
  8. 2016End to End Learning for Self-Driving Carscited by 1,000
  9. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
  10. 2017End-to-end Driving via Conditional Imitation Learningcited by 479
  11. 2018ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worstcited by 391
  12. 2019A Survey of Autonomous Driving: Common Practices and Emerging Technologiescited by 373
31 more
  1. 2019Learning by Cheatingcited by 361
  2. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 343
  3. 2021DenseTNT: End-to-end Trajectory Prediction from Dense Goal Setscited by 292
  4. 2021FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Camerascited by 239
  5. 2020Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Netcited by 228
  6. 2021MP3: A Unified Model to Map, Perceive, Predict and Plancited by 211
  7. 2021End-to-End Urban Driving by Imitating a Reinforcement Learning Coachcited by 191
  8. 2020Predicting Semantic Map Representations from Images using Pyramid Occupancy Networkscited by 175
  9. 2021NEAT: Neural Attention Fields for End-to-End Autonomous Drivingcited by 174
  10. 2019Cross-view Semantic Segmentation for Sensing Surroundingscited by 173
  11. 2020Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representationscited by 150
  12. 2020Video Panoptic Segmentationcited by 141
  13. 2022PersFormer: 3D Lane Detection via Perspective Transformer and the OpenLane Benchmarkcited by 140
  14. 2022Learning from All Vehiclescited by 135
  15. 2021Learning to drive from a world on railscited by 114
  16. 2020MotionNet: Joint Perception and Motion Prediction for Autonomous Driving Based on Bird's Eye View Mapscited by 100
  17. 2021LookOut: Diverse Multi-Future Prediction and Planning for Self-Drivingcited by 96
  18. 2018Uncertainty-aware Short-term Motion Prediction of Traffic Actors for Autonomous Drivingcited by 90
  19. 2018Monocular Semantic Occupancy Grid Mapping with Convolutional Variational Encoder-Decoder Networkscited by 84
  20. 2020TPNet: Trajectory Proposal Network for Motion Predictioncited by 80
  21. 2020DSDNet: Deep Structured self-Driving Networkcited by 61
  22. 2021GRI: General Reinforced Imitation and its Application to Vision-Based Autonomous Drivingcited by 59
  23. 2021Fine-grained Semantics-aware Representation Enhancement for Self-supervised Monocular Depth Estimationcited by 55
  24. 2019Jointly Learnable Behavior and Trajectory Planning for Self-Driving Vehiclescited by 51
  25. 2020Probabilistic Future Prediction for Video Scene Understandingcited by 45
  26. 2020Multi-View Fusion of Sensor Data for Improved Perception and Prediction in Autonomous Drivingcited by 41
  27. 2020BEV-Seg: Bird's Eye View Semantic Segmentation Using Geometry and Semantic Point Cloudcited by 38
  28. 2016From Perception to Decision: A Data-driven Approach to End-to-end Motion Planning for Autonomous Ground Robotscited by 37
  29. 2020Driving among Flatmobiles: Bird-Eye-View occupancy grids from a monocular camera for holistic trajectory planningcited by 24
  30. 2020Perceive, Attend, and Drive: Learning Spatial Attention for Safe Self-Drivingcited by 24
  31. 2019Affordance Learning for End-to-End Visuomotor Robot Controlcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 48; 2024: 86; 2025: 134; 2026: 93.

365 papers.

Sort:

By area:

  1. 2022Planning-oriented Autonomous Drivingcited by 1,068
  2. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  3. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  4. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  5. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  6. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  7. 2024NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarkingcited by 343
  8. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  9. 2024DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Drivingcited by 324
  10. 2024VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planningcited by 287
  11. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 258
  12. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuningcited by 252
38 more
  1. 2024Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Drivingcited by 250
  2. 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
  3. 2024SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representationcited by 239
  4. 2023Parting with Misconceptions about Learning-based Vehicle Motion Planningcited by 238
  5. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  6. 2023OccWorld: Learning a 3D Occupancy World Model for Autonomous Drivingcited by 230
  7. 2024GenAD: Generative End-to-End Autonomous Drivingcited by 225
  8. 2023Scene as Occupancycited by 208
  9. 2023Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenescited by 191
  10. 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Drivingcited by 190
  11. 2025ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generationcited by 177
  12. 2025OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Modelcited by 172
  13. 2023Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Drivingcited by 168
  14. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 165
  15. 2023A Language Agent for Autonomous Drivingcited by 161
  16. 2024Enhancing End-to-End Autonomous Driving with Latent World Modelcited by 161
  17. 2024Behavior Generation with Latent Actionscited by 159
  18. 2024Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Drivingcited by 156
  19. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  20. 2025SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignmentcited by 149
  21. 2025End-to-End Driving with Online Trajectory Evaluation via BEV World Modelcited by 145
  22. 2024VLP: Vision Language Planning for Autonomous Drivingcited by 125
  23. 2025DriveTransformer: Unified Transformer for Scalable End-to-End Autonomous Drivingcited by 125
  24. 2024DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representationcited by 123
  25. 2022Delving into the Devils of Bird's-eye-view Perception: A Review, Evaluation and Recipecited by 122
  26. 2025Epona: Autoregressive Diffusion World Model for Autonomous Drivingcited by 115
  27. 2024textitS3Gaussian: Self-Supervised Street Gaussians for Autonomous Drivingcited by 105
  28. 2024Towards learning-based planning:The nuPlan benchmark for real-world autonomous drivingcited by 103
  29. 2023ADriver-I: A General World Model for Autonomous Drivingcited by 102
  30. 2023Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Surveycited by 101
  31. 2024DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Drivingcited by 98
  32. 2023Visual Point Cloud Forecasting enables Scalable Autonomous Drivingcited by 97
  33. 2024DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformerscited by 96
  34. 2025Hydra-MDP++: Advancing End-to-End Driving via Expert-Guided Hydra-Distillationcited by 93
  35. 2023ReasonNet: End-to-End Driving with Temporal and Global Reasoningcited by 92
  36. 2025DriveSuprim: Towards Precise Trajectory Selection for End-to-End Planningcited by 91
  37. 2025World4Drive: End-to-End Autonomous Driving via Intention-aware Physical Latent World Modelcited by 91
  38. 2024ReconDreamer: Crafting World Models for Driving Scene Reconstruction via Online Restorationcited by 89

50 most cited of 365 All 365 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Hierarchical End-to-End Autonomous Driving: Integrating BEV Perception with Deep Reinforcement Learning
  2. 2024GaussianAD: Gaussian-Centric End-to-End Autonomous Driving
  3. 2026Explicit Geometric Chain-of-Thought for Vision-Language-Action in Autonomous Driving
  4. 2025ME3-BEV: Mamba-Enhanced Deep Reinforcement Learning for End-to-End Autonomous Driving with BEV-Perception
  5. 2025Future-Aware End-to-End Driving: Bidirectional Modeling of Trajectory Planning and Scene Evolution
  6. 2026Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Driving
  7. 2023Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Models
  8. 2025Learning-based 3D Reconstruction in Autonomous Driving: A Comprehensive Survey