mlsanityData to 14 Oct 2026

STELLAR: Scaling 3D Perception Large Models for Autonomous Driving

arXiv 2605.20390 ↗First posted 19 May 2026cs.CV cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Model scaling has demonstrated remarkable success through large-scale training on diverse datasets. It remains an open question whether the same paradigm would apply to autonomous driving perception systems due to unique challenges, such as fusing heterogeneous sensor data and the need for sophisticated 3D spatial understanding. To bridge this gap, we present a comprehensive study on systematically analyzing the impact of scale on these systems. We develop our STELLAR model based on Sparse Window Transformer, by extending the input modalities to include LiDAR, radar, camera, and map prior. We train the model on a large-scale dataset of 50 million driving examples with up to 500 million parameters. Our large-scale experiments reveal empirical scaling trends that connect model performance to model size, data, and compute. The resulting model establishes a new state-of-the-art on the Waymo Open Dataset challenge, outperforming prior arts by a large margin. Our work demonstrates that large-scale training is a highly promising path for advancing the capabilities of perception models for autonomous driving.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 55 of its 65 references that are in the map. First posted on arXiv 19 May 2026.

Builds on

55 of 65 references.

References · 55

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  8. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  9. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  10. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  11. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,715
  12. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
38 more
  1. 2020Center-based 3D Object Detection and Trackingcited by 1,144
  2. 2022Planning-oriented Autonomous Drivingcited by 1,087
  3. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,044
  4. 2016Deep Networks with Stochastic Depthcited by 979
  5. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 850
  6. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  7. 2021BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-Viewcited by 713
  8. 2023Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecastingcited by 707
  9. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  10. 2019CenterNet: Keypoint Triplets for Object Detectioncited by 654
  11. 2023Point Transformer V3: Simpler, Faster, Strongercited by 589
  12. 2018Rethinking ImageNet Pre-trainingcited by 377
  13. 2023Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Drivingcited by 332
  14. 2023SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Drivingcited by 316
  15. 2020Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Netcited by 228
  16. 2021Embracing Single Stride 3D Object Detector with Sparse Transformercited by 193
  17. 2019End-to-End Multi-View Fusion for 3D Object Detection in LiDAR Point Cloudscited by 171
  18. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 170
  19. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 127
  20. 2024Towards learning-based planning:The nuPlan benchmark for real-world autonomous drivingcited by 104
  21. 2022CenterFormer: Center-based Transformer for 3D Object Detectioncited by 99
  22. 2022SWFormer: Sparse Window Transformer for 3D Object Detection in Point Cloudscited by 92
  23. 2022Fully Sparse 3D Object Detectioncited by 84
  24. 2023LoGoNet: Towards Accurate 3D Object Detection with Local-to-Global Cross-Modal Fusioncited by 80
  25. 2023UniPAD: A Universal Pre-training Paradigm for Autonomous Drivingcited by 73
  26. 2020SurfelGAN: Synthesizing Realistic Sensor Data for Autonomous Drivingcited by 68
  27. 2024SAFDNet: A Simple and Effective Network for Fully Sparse 3D Object Detectioncited by 52
  28. 2023HEDNet: A Hierarchical Encoder-Decoder Network for 3D Object Detection in Point Cloudscited by 45
  29. 2024UnO: Unsupervised Occupancy Fields for Perception and Forecastingcited by 41
  30. 2022Masked Autoencoder for Self-Supervised Pre-training on Lidar Point Cloudscited by 39
  31. 2023The Impact of Depth on Compositional Generalization in Transformer Language Modelscited by 38
  32. 2024LION: Linear Group RNN for 3D Object Detection in Point Cloudscited by 38
  33. 2023FSD V2: Improving Fully Sparse 3D Object Detection with Virtual Voxelscited by 34
  34. 2023Super Sparse 3D Object Detectioncited by 32
  35. 2024DriveGPT: Scaling Autoregressive Behavior Models for Drivingcited by 29
  36. 2024SEED: A Simple and Effective 3D DETR in Point Cloudscited by 19
  37. 2025Scaling Laws of Motion Forecasting and Planning -- Technical Reportcited by 19
  38. 2023MSF: Motion-guided Sequential Fusion for Efficient 3D Object Detection from Point Cloud Sequencescited by 15

50 most cited of 55 All 55 (JSON)

Built on by

Not cited yet.