# UniPAD: A Universal Pre-training Paradigm for Autonomous Driving

arXiv: https://arxiv.org/abs/2310.08370 · HTML: https://mlsanity.com/paper/2310.08370/

## Key facts

- Cited by 73 papers in the map (data through 14 Oct 2026).
- Builds on 97 of its 113 references that are in the map.
- First posted on arXiv 12 Oct 2023.

## Builds on (most cited first, 30 of 97)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [TensoRF: Tensorial Radiance Fields](https://mlsanity.com/paper/2203.09517/) (2022)
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022)
- [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020)
- [Volume Rendering of Neural Implicit Surfaces](https://mlsanity.com/paper/2106.12052/) (2021)
- [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019)
- [BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation](https://mlsanity.com/paper/2205.13542/) (2022)
- [BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-View](https://mlsanity.com/paper/2112.11790/) (2021)
- [Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling](https://mlsanity.com/paper/2111.14819/) (2021)
- [UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction](https://mlsanity.com/paper/2104.10078/) (2021)
- [BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection](https://mlsanity.com/paper/2206.10092/) (2022)
- [PETR: Position Embedding Transformation for Multi-View 3D Object Detection](https://mlsanity.com/paper/2203.05625/) (2022)
- [PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding](https://mlsanity.com/paper/2007.10985/) (2020)
- [Point-NeRF: Point-based Neural Radiance Fields](https://mlsanity.com/paper/2201.08845/) (2022)

## Built on by (most cited first, 30 of 73)

- [Visual Point Cloud Forecasting enables Scalable Autonomous Driving](https://mlsanity.com/paper/2312.17655/) (2023)
- [Aether: Geometric-Aware Unified World Modeling](https://mlsanity.com/paper/2503.18945/) (2025)
- [DriveWorld: 4D Pre-trained Scene Understanding via World Models for Autonomous Driving](https://mlsanity.com/paper/2405.04390/) (2024)
- [A Survey of World Models for Autonomous Driving](https://mlsanity.com/paper/2501.11260/) (2025)
- [HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generation](https://mlsanity.com/paper/2501.14729/) (2025)
- [DiffusionDriveV2: Reinforcement Learning-Constrained Truncated Diffusion Modeling in End-to-End Autonomous Driving](https://mlsanity.com/paper/2512.07745/) (2025)
- [DrivingForward: Feed-forward 3D Gaussian Splatting for Driving Scene Reconstruction from Flexible Surround-view Input](https://mlsanity.com/paper/2409.12753/) (2024)
- [Point Cloud Matters: Rethinking the Impact of Different Observation Spaces on Robot Learning](https://mlsanity.com/paper/2402.02500/) (2024)
- [OA-CNNs: Omni-Adaptive Sparse CNNs for 3D Semantic Segmentation](https://mlsanity.com/paper/2403.14418/) (2024)
- [UnO: Unsupervised Occupancy Fields for Perception and Forecasting](https://mlsanity.com/paper/2406.08691/) (2024)
- [DeepVerse: 4D Autoregressive Video Generation as a World Model](https://mlsanity.com/paper/2506.01103/) (2025)
- [FantasyWorld: Geometry-Consistent World Modeling via Unified Video and 3D Prediction](https://mlsanity.com/paper/2509.21657/) (2025)
- [DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by Distilling Neural Fields and Foundation Model Features](https://mlsanity.com/paper/2406.12095/) (2024)
- [SPA: 3D Spatial-Awareness Enables Effective Embodied Representation](https://mlsanity.com/paper/2410.08208/) (2024)
- [GroupContrast: Semantic-aware Self-supervised Representation Learning for 3D Understanding](https://mlsanity.com/paper/2403.09639/) (2024)
- [Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectives](https://mlsanity.com/paper/2402.02968/) (2024)
- [Hulk: A Universal Knowledge Translator for Human-Centric Tasks](https://mlsanity.com/paper/2312.01697/) (2023)
- [EVolSplat: Efficient Volume-based Gaussian Splatting for Urban View Synthesis](https://mlsanity.com/paper/2503.20168/) (2025)
- [Neural Radiance Field in Autonomous Driving: A Survey](https://mlsanity.com/paper/2404.13816/) (2024)
- [OccFeat: Self-supervised Occupancy Feature Prediction for Pretraining BEV Segmentation Networks](https://mlsanity.com/paper/2404.14027/) (2024)
- [GGS: Generalizable Gaussian Splatting for Lane Switching in Autonomous Driving](https://mlsanity.com/paper/2409.02382/) (2024)
- [VisionPAD: A Vision-Centric Pre-training Paradigm for Autonomous Driving](https://mlsanity.com/paper/2411.14716/) (2024)
- [DriveGen3D: Boosting Feed-Forward Driving Scene Generation with Efficient Video Diffusion](https://mlsanity.com/paper/2510.15264/) (2025)
- [Applications of Large Scale Foundation Models for Autonomous Driving](https://mlsanity.com/paper/2311.12144/) (2023)
- [AnoVox: A Benchmark for Multimodal Anomaly Detection in Autonomous Driving](https://mlsanity.com/paper/2405.07865/) (2024)
- [X-Drive: Cross-modality consistent multi-sensor data synthesis for driving scenarios](https://mlsanity.com/paper/2411.01123/) (2024)
- [Masked Modeling for Self-supervised Representation Learning on Vision and Beyond](https://mlsanity.com/paper/2401.00897/) (2023)
- [GaussianPretrain: A Simple Unified 3D Gaussian Representation for Visual Pre-training in Autonomous Driving](https://mlsanity.com/paper/2411.12452/) (2024)
- [Learning-based 3D Reconstruction in Autonomous Driving: A Comprehensive Survey](https://mlsanity.com/paper/2503.14537/) (2025)
- [TurboTrain: Towards Efficient and Balanced Multi-Task Learning for Multi-Agent Perception and Prediction](https://mlsanity.com/paper/2508.04682/) (2025)

## Abstract (arXiv)

In the context of autonomous driving, the significance of effective feature learning is widely acknowledged. While conventional 3D self-supervised pre-training methods have shown widespread success, most methods follow the ideas originally designed for 2D images. In this paper, we present UniPAD, a novel self-supervised learning paradigm applying 3D volumetric differentiable rendering. UniPAD implicitly encodes 3D space, facilitating the reconstruction of continuous 3D shape structures and the intricate appearance characteristics of their 2D projections. The flexibility of our method enables seamless integration into both 2D and 3D frameworks, enabling a more holistic comprehension of the scenes. We manifest the feasibility and effectiveness of UniPAD by conducting extensive experiments on various downstream 3D tasks. Our method significantly improves lidar-, camera-, and lidar-camera-based baseline by 9.1, 7.7, and 6.9 NDS, respectively. Notably, our pre-training pipeline achieves 73.2 NDS for 3D object detection and 79.4 mIoU for 3D semantic segmentation on the nuScenes validation set, achieving state-of-the-art results in comparison with previous methods. The code will be available at https://github.com/Nightmare-n/UniPAD.