# MV-DeepSDF: Implicit Modeling with Multi-Sweep Point Clouds for 3D Vehicle Reconstruction in Autonomous Driving

arXiv: https://arxiv.org/abs/2309.16715 · HTML: https://mlsanity.com/paper/2309.16715/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 54 references that are in the map.
- First posted on arXiv 21 Aug 2023.

## Builds on (most cited first, 30 of 40)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021)
- [Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images](https://mlsanity.com/paper/1804.01654/) (2018)
- [AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generation](https://mlsanity.com/paper/1802.05384/) (2018)
- [Point-NeRF: Point-based Neural Radiance Fields](https://mlsanity.com/paper/2201.08845/) (2022)
- [PCN: Point Completion Network](https://mlsanity.com/paper/1808.00671/) (2018)
- [Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstruction](https://mlsanity.com/paper/2003.10983/) (2020)
- [Neural RGB-D Surface Reconstruction](https://mlsanity.com/paper/2104.04532/) (2021)
- [PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformers](https://mlsanity.com/paper/2108.08839/) (2021)
- [Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesis](https://mlsanity.com/paper/1612.00101/) (2016)
- [GRNet: Gridding Residual Network for Dense Point Cloud Completion](https://mlsanity.com/paper/2006.03761/) (2020)
- [Semantic3D.net: A new Large-scale Point Cloud Classification Benchmark](https://mlsanity.com/paper/1704.03847/) (2017)
- [SG-NN: Sparse Generative Neural Networks for Self-Supervised Scene Completion of RGB-D Scans](https://mlsanity.com/paper/1912.00036/) (2019)
- [AdaPoinTr: Diverse Point Cloud Completion with Adaptive Geometry-Aware Transformers](https://mlsanity.com/paper/2301.04545/) (2023)
- [Curriculum DeepSDF](https://mlsanity.com/paper/2003.08593/) (2020)
- [Neural Surface Reconstruction of Dynamic Scenes with Monocular RGB-D Camera](https://mlsanity.com/paper/2206.15258/) (2022)
- [Multi-view Consistency as Supervisory Signal for Learning Shape and Pose Prediction](https://mlsanity.com/paper/1801.03910/) (2018)
- [Multi-view 3D Reconstruction with Transformer](https://mlsanity.com/paper/2103.12957/) (2021)
- [GIFS: Neural Implicit Function for General Shape Representation](https://mlsanity.com/paper/2204.07126/) (2022)
- [Weakly-supervised 3D Shape Completion in the Wild](https://mlsanity.com/paper/2008.09110/) (2020)
- [NeAT: Learning Neural Implicit Surfaces with Arbitrary Topologies from Multi-view Images](https://mlsanity.com/paper/2303.12012/) (2023)

## Built on by (most cited first, 11 of 11)

- [UniGaussian: Driving Scene Reconstruction from Multiple Camera Models via Unified Gaussian Representations](https://mlsanity.com/paper/2411.15355/) (2024)
- [VQA-Diff: Exploiting VQA and Diffusion for Zero-Shot Image-to-3D Vehicle Asset Generation in Autonomous Driving](https://mlsanity.com/paper/2407.06516/) (2024)
- [HIPPo: Harnessing Image-to-3D Priors for Model-free Zero-shot 6D Pose Estimation](https://mlsanity.com/paper/2502.10606/) (2025)
- [Learning Effective NeRFs and SDFs Representations with 3D Generative Adversarial Networks for 3D Object Generation](https://mlsanity.com/paper/2309.16110/) (2023)
- [ArmGS: Composite Gaussian Appearance Refinement for Modeling Dynamic Urban Environments](https://mlsanity.com/paper/2507.03886/) (2025)
- [L-PR: Exploiting LiDAR Fiducial Marker for Unordered Low Overlap Multiview Point Cloud Registration](https://mlsanity.com/paper/2406.03298/) (2024)
- [Uni-Render: A Unified Accelerator for Real-Time Rendering Across Diverse Neural Renderers](https://mlsanity.com/paper/2503.23644/) (2025)
- [Efficient 3D Perception on Multi-Sweep Point Cloud with Gumbel Spatial Pruning](https://mlsanity.com/paper/2411.07742/) (2024)
- [NF-SLAM: Effective, Normalizing Flow-supported Neural Field representations for object-level visual SLAM in automotive applications](https://mlsanity.com/paper/2503.11199/) (2025)
- [Implicit 3D scene reconstruction using deep learning towards efficient collision understanding in autonomous driving](https://mlsanity.com/paper/2506.15806/) (2025)
- [SI-Diff: A Framework for Learning Search and High-Precision Insertion with a Force-Domain Diffusion Policy](https://mlsanity.com/paper/2605.12247/) (2026)

## Abstract (arXiv)

Reconstructing 3D vehicles from noisy and sparse partial point clouds is of great significance to autonomous driving. Most existing 3D reconstruction methods cannot be directly applied to this problem because they are elaborately designed to deal with dense inputs with trivial noise. In this work, we propose a novel framework, dubbed MV-DeepSDF, which estimates the optimal Signed Distance Function (SDF) shape representation from multi-sweep point clouds to reconstruct vehicles in the wild. Although there have been some SDF-based implicit modeling methods, they only focus on single-view-based reconstruction, resulting in low fidelity. In contrast, we first analyze multi-sweep consistency and complementarity in the latent feature space and propose to transform the implicit space shape estimation problem into an element-to-set feature extraction problem. Then, we devise a new architecture to extract individual element-level representations and aggregate them to generate a set-level predicted latent code. This set-level latent code is an expression of the optimal 3D shape in the implicit space, and can be subsequently decoded to a continuous SDF of the vehicle. In this way, our approach learns consistent and complementary information among multi-sweeps for 3D vehicle reconstruction. We conduct thorough experiments on two real-world autonomous driving datasets (Waymo and KITTI) to demonstrate the superiority of our approach over state-of-the-art alternative methods both qualitatively and quantitatively.