# SSPU-Net: Self-Supervised Point Cloud Upsampling via Differentiable Rendering

arXiv: https://arxiv.org/abs/2108.00454 · HTML: https://mlsanity.com/paper/2108.00454/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 62 references that are in the map.
- First posted on arXiv 1 Aug 2021.

## Builds on (most cited first, 30 of 34)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flows](https://mlsanity.com/paper/1906.12320/) (2019)
- [Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving](https://mlsanity.com/paper/1812.07179/) (2018)
- [Neural 3D Mesh Renderer](https://mlsanity.com/paper/1711.07566/) (2017)
- [Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://mlsanity.com/paper/1904.01786/) (2019)
- [PCN: Point Completion Network](https://mlsanity.com/paper/1808.00671/) (2018)
- [PU-Net: Point Cloud Upsampling Network](https://mlsanity.com/paper/1801.06761/) (2018)
- [PointNetVLAD: Deep Point Cloud Based Retrieval for Large-Scale Place Recognition](https://mlsanity.com/paper/1804.03492/) (2018)
- [3D Point Cloud Generative Adversarial Network Based on Tree Structured Graph Convolutions](https://mlsanity.com/paper/1905.06292/) (2019)
- [PU-GAN: a Point Cloud Upsampling Adversarial Network](https://mlsanity.com/paper/1907.10844/) (2019)
- [MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction](https://mlsanity.com/paper/1703.10580/) (2017)
- [Nonlinear 3D Face Morphable Model](https://mlsanity.com/paper/1804.03786/) (2018)
- [Adversarial Sensor Attack on LiDAR-based Perception in Autonomous Driving](https://mlsanity.com/paper/1907.06826/) (2019)
- [Single-Image SVBRDF Capture with a Rendering-Aware Deep Network](https://mlsanity.com/paper/1810.09718/) (2018)
- [Patch-based Progressive 3D Point Set Upsampling](https://mlsanity.com/paper/1811.11286/) (2018)
- [Unsupervised Training for 3D Morphable Model Regression](https://mlsanity.com/paper/1806.06098/) (2018)
- [Self-supervised Multi-level Face Model Learning for Monocular Reconstruction at over 250 Hz](https://mlsanity.com/paper/1712.02859/) (2017)
- [Interpolated Convolutional Networks for 3D Point Cloud Understanding](https://mlsanity.com/paper/1908.04512/) (2019)
- [EC-Net: an Edge-aware Point set Consolidation Network](https://mlsanity.com/paper/1807.06010/) (2018)
- [PUGeo-Net: A Geometry-centric Network for 3D Point Cloud Upsampling](https://mlsanity.com/paper/2002.10277/) (2020)
- [A*3D Dataset: Towards Autonomous Driving in Challenging Environments](https://mlsanity.com/paper/1909.07541/) (2019)
- [SSPC-Net: Semi-supervised Semantic 3D Point Cloud Segmentation Network](https://mlsanity.com/paper/2104.07861/) (2021)
- [Progressive Point Cloud Deconvolution Generation Network](https://mlsanity.com/paper/2007.05361/) (2020)
- [FlowFusion: Dynamic Dense RGB-D SLAM Based on Optical Flow](https://mlsanity.com/paper/2003.05102/) (2020)
- [Meta-PU: An Arbitrary-Scale Upsampling Network for Point Cloud](https://mlsanity.com/paper/2102.04317/) (2021)
- [Efficient 3D Point Cloud Feature Learning for Large-Scale Place Recognition](https://mlsanity.com/paper/2101.02374/) (2021)
- [Material Editing Using a Physically Based Rendering Network](https://mlsanity.com/paper/1708.00106/) (2017)

## Built on by (most cited first, 17 of 17)

- [Neural Points: Point Cloud Representation with Neural Fields for Arbitrary Upsampling](https://mlsanity.com/paper/2112.04148/) (2021)
- [PU-Transformer: Point Cloud Upsampling Transformer](https://mlsanity.com/paper/2111.12242/) (2021)
- [Self-Supervised Arbitrary-Scale Point Clouds Upsampling via Implicit Neural Representation](https://mlsanity.com/paper/2204.08196/) (2022)
- [P2C: Self-Supervised Point Cloud Completion from Single Partial Clouds](https://mlsanity.com/paper/2307.14726/) (2023)
- [Point Cloud Upsampling via Cascaded Refinement Network](https://mlsanity.com/paper/2210.03942/) (2022)
- [TULIP: Transformer for Upsampling of LiDAR Point Clouds](https://mlsanity.com/paper/2312.06733/) (2023)
- [PC2-PU: Patch Correlation and Point Correlation for Effective Point Cloud Upsampling](https://mlsanity.com/paper/2109.09337/) (2021)
- [IPDAE: Improved Patch-Based Deep Autoencoder for Lossy Point Cloud Geometry Compression](https://mlsanity.com/paper/2208.02519/) (2022)
- [Arbitrary-Scale Point Cloud Upsampling by Voxel-Based Network with Latent Geometric-Consistent Learning](https://mlsanity.com/paper/2403.05117/) (2024)
- [Deep Learning for 3D Point Cloud Enhancement: A Survey](https://mlsanity.com/paper/2411.00857/) (2024)
- [Arbitrary point cloud upsampling via Dual Back-Projection Network](https://mlsanity.com/paper/2307.08992/) (2023)
- [PU-Ray: Domain-Independent Point Cloud Upsampling via Ray Marching on Neural Implicit Surface](https://mlsanity.com/paper/2310.08755/) (2023)
- [MBPU: A Plug-and-Play State Space Model for Point Cloud Upsamping with Fast Point Rendering](https://mlsanity.com/paper/2410.15941/) (2024)
- [SPU-IMR: Self-supervised Arbitrary-scale Point Cloud Upsampling via Iterative Mask-recovery Network](https://mlsanity.com/paper/2502.19452/) (2025)
- [Self-Supervised Learning for Point Clouds Data: A Survey](https://mlsanity.com/paper/2305.11881/) (2023)
- [MMAPS: End-to-End Multi-Grained Multi-Modal Attribute-Aware Product Summarization](https://mlsanity.com/paper/2308.11351/) (2023)
- [Preserving Topological and Geometric Embeddings for Point Cloud Recovery](https://mlsanity.com/paper/2507.19121/) (2025)

## Abstract (arXiv)

Point clouds obtained from 3D sensors are usually sparse. Existing methods mainly focus on upsampling sparse point clouds in a supervised manner by using dense ground truth point clouds. In this paper, we propose a self-supervised point cloud upsampling network (SSPU-Net) to generate dense point clouds without using ground truth. To achieve this, we exploit the consistency between the input sparse point cloud and generated dense point cloud for the shapes and rendered images. Specifically, we first propose a neighbor expansion unit (NEU) to upsample the sparse point clouds, where the local geometric structures of the sparse point clouds are exploited to learn weights for point interpolation. Then, we develop a differentiable point cloud rendering unit (DRU) as an end-to-end module in our network to render the point cloud into multi-view images. Finally, we formulate a shape-consistent loss and an image-consistent loss to train the network so that the shapes of the sparse and dense point clouds are as consistent as possible. Extensive results on the CAD and scanned datasets demonstrate that our method can achieve impressive results in a self-supervised manner. Code is available at https://github.com/fpthink/SSPU-Net.