mlsanityData to 7 Oct 2026

Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D

arXiv 2008.05711 ↗First posted 13 Aug 2020cs.CVcited by 1,040

Not on the map. Why?

Cited by 121 in End-to-end autonomous driving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The goal of perception for autonomous vehicles is to extract semantic representations from multiple sensors and fuse these representations into a single "bird's-eye-view" coordinate frame for consumption by motion planning. We propose a new end-to-end architecture that directly extracts a bird's-eye-view representation of a scene given image data from an arbitrary number of cameras. The core idea behind our approach is to "lift" each image individually into a frustum of features for each camera, then "splat" all frustums into a rasterized bird's-eye-view grid. By training on the entire camera rig, we provide evidence that our model is able to learn not only how to represent images but how to fuse predictions from all cameras into a single cohesive representation of the scene while being robust to calibration error. On standard bird's-eye-view tasks such as object segmentation and map segmentation, our model outperforms all baselines and prior work. In pursuit of the goal of learning dense representations for motion planning, we show that the representations inferred by our model enable interpretable end-to-end motion planning by "shooting" template trajectories into a bird's-eye-view cost map output by our network. We benchmark our approach against models that use oracle depth from lidar. Project page with code: https://nv-tlabs.github.io/lift-splat-shoot .

Key facts. Cited by 1,040 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Aug 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 27; 2022: 108; 2023: 212; 2024: 279; 2025: 241; 2026: 173.

1,040 papers.

Sort:

By area:

  1. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  2. 2022Planning-oriented Autonomous Drivingcited by 1,068
  3. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 844
  4. 2021BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-Viewcited by 711
  5. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  6. 2022BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detectioncited by 568
  7. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 550
  8. 2022TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformerscited by 464
  9. 2023Tri-Perspective View for Vision-Based 3D Semantic Occupancy Predictioncited by 404
  10. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  11. 2022PETRv2: A Unified Framework for 3D Perception from Multi-Camera Imagescited by 351
  12. 2022BEVDet4D: Exploit Temporal Cues in Multi-camera 3D Object Detectioncited by 340
38 more
  1. 2022BEVFusion: A Simple and Robust LiDAR-Camera Fusion Frameworkcited by 334
  2. 2021Categorical Depth Distribution Network for Monocular 3D Object Detectioncited by 326
  3. 2023Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Drivingcited by 325
  4. 2022MapTR: Structured Modeling and Learning for Online Vectorized HD Map Constructioncited by 311
  5. 2023SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Drivingcited by 310
  6. 2022BEVFormer v2: Adapting Modern Image Backbones to Bird's-Eye-View Recognition via Perspective Supervisioncited by 302
  7. 2023OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Predictioncited by 287
  8. 2024VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planningcited by 287
  9. 2021HDMapNet: An Online HD Map Construction and Evaluation Frameworkcited by 280
  10. 2022Cross-view Transformers for real-time Map-view Semantic Segmentationcited by 264
  11. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
  12. 2022VectorMapNet: End-to-end Vectorized HD Map Learningcited by 243
  13. 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
  14. 2021FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Camerascited by 239
  15. 2024SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representationcited by 239
  16. 2021PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detectioncited by 232
  17. 2023OccWorld: Learning a 3D Occupancy World Model for Autonomous Drivingcited by 230
  18. 2022CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformerscited by 227
  19. 2024GenAD: Generative End-to-End Autonomous Drivingcited by 225
  20. 2021MP3: A Unified Model to Map, Perceive, Predict and Plancited by 211
  21. 2022Unifying Voxel-based Representation with Transformer for 3D Object Detectioncited by 202
  22. 2023MapTRv2: An End-to-End Framework for Online Vectorized HD Map Constructioncited by 192
  23. 2022BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Drivingcited by 191
  24. 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Drivingcited by 190
  25. 2022Time Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detectioncited by 177
  26. 2025ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generationcited by 177
  27. 2021NEAT: Neural Attention Fields for End-to-End Autonomous Drivingcited by 174
  28. 2023Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Drivingcited by 168
  29. 2023FB-OCC: 3D Occupancy Prediction based on Forward-Backward View Transformationcited by 162
  30. 2022Sparse4D: Multi-view 3D Object Detection with Sparse Spatial-Temporal Fusioncited by 161
  31. 2024Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Drivingcited by 156
  32. 2022M2BEV: Multi-Camera Joint 3D Detection and Segmentation with Unified Birds-Eye View Representationcited by 155
  33. 2024GaussianFormer: Scene as Gaussians for Vision-Based 3D Semantic Occupancy Predictioncited by 155
  34. 2022Model-Based Imitation Learning for Urban Drivingcited by 149
  35. 2023SparseBEV: High-Performance Sparse 3D Object Detection from Multi-Camera Videoscited by 149
  36. 2023DDP: Diffusion Model for Dense Visual Predictioncited by 148
  37. 2022PolarFormer: Multi-camera 3D Object Detection with Polar Transformercited by 147
  38. 2023StreamMapNet: Streaming Mapping Network for Vectorized Online HD Map Constructioncited by 143

50 most cited of 1,040 All 1,040 (JSON)