# SE-MD: A Single-encoder multiple-decoder deep network for point cloud generation from 2D images

arXiv: https://arxiv.org/abs/2106.15325 · HTML: https://mlsanity.com/paper/2106.15325/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 44 references that are in the map.
- First posted on arXiv 17 Jun 2021.

## Builds on (most cited first, 30 of 31)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images](https://mlsanity.com/paper/1804.01654/) (2018)
- [3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction](https://mlsanity.com/paper/1604.00449/) (2016)
- [Deep Learning for 3D Point Clouds: A Survey](https://mlsanity.com/paper/1912.12033/) (2019)
- [Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://mlsanity.com/paper/1904.01786/) (2019)
- [PCN: Point Completion Network](https://mlsanity.com/paper/1808.00671/) (2018)
- [From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Network](https://mlsanity.com/paper/1907.03670/) (2019)
- [Learning Category-Specific Mesh Reconstruction from Image Collections](https://mlsanity.com/paper/1803.07549/) (2018)
- [Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesis](https://mlsanity.com/paper/1612.00101/) (2016)
- [Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling](https://mlsanity.com/paper/1804.04610/) (2018)
- [Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs](https://mlsanity.com/paper/1703.09438/) (2017)
- [Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction](https://mlsanity.com/paper/1706.07036/) (2017)
- [Visual Object Networks: Image Generation with Disentangled 3D Representation](https://mlsanity.com/paper/1812.02725/) (2018)
- [Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Review](https://mlsanity.com/paper/2004.05224/) (2020)
- [Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision](https://mlsanity.com/paper/1612.00814/) (2016)
- [Hierarchical Surface Prediction for 3D Object Reconstruction](https://mlsanity.com/paper/1704.00710/) (2017)
- [Learning to Reconstruct Shapes from Unseen Classes](https://mlsanity.com/paper/1812.11166/) (2018)
- [Multi-view 3D Models from Single Images with a Convolutional Network](https://mlsanity.com/paper/1511.06702/) (2015)
- [Multi-view Consistency as Supervisory Signal for Learning Shape and Pose Prediction](https://mlsanity.com/paper/1801.03910/) (2018)
- [3D-LMNet: Latent Embedding Matching for Accurate and Diverse 3D Point Cloud Reconstruction from a Single Image](https://mlsanity.com/paper/1807.07796/) (2018)
- [Dense 3D Point Cloud Reconstruction Using a Deep Pyramid Network](https://mlsanity.com/paper/1901.08906/) (2019)
- [Multiview Aggregation for Learning Category-Specific Shape Reconstruction](https://mlsanity.com/paper/1907.01085/) (2019)
- [Inferring Point Clouds from Single Monocular Images by Depth Intermediation](https://mlsanity.com/paper/1812.01402/) (2018)
- [Learning to Generate Dense Point Clouds with Textures on Multiple Categories](https://mlsanity.com/paper/1912.10545/) (2019)
- [Silhouette Guided Point Cloud Reconstruction beyond Occlusion](https://mlsanity.com/paper/1907.12253/) (2019)

## Abstract (arXiv)

3D model generation from single 2D RGB images is a challenging and actively researched computer vision task. Various techniques using conventional network architectures have been proposed for the same. However, the body of research work is limited and there are various issues like using inefficient 3D representation formats, weak 3D model generation backbones, inability to generate dense point clouds, dependence of post-processing for generation of dense point clouds, and dependence on silhouettes in RGB images. In this paper, a novel 2D RGB image to point cloud conversion technique is proposed, which improves the state of art in the field due to its efficient, robust and simple model by using the concept of parallelization in network architecture. It not only uses the efficient and rich 3D representation of point clouds, but also uses a novel and robust point cloud generation backbone in order to address the prevalent issues. This involves using a single-encoder multiple-decoder deep network architecture wherein each decoder generates certain fixed viewpoints. This is followed by fusing all the viewpoints to generate a dense point cloud. Various experiments are conducted on the technique and its performance is compared with those of other state of the art techniques and impressive gains in performance are demonstrated. Code is available at https://github.com/mueedhafiz1982/