# Deep Hybrid Self-Prior for Full 3D Mesh Generation

arXiv: https://arxiv.org/abs/2108.08017 · HTML: https://mlsanity.com/paper/2108.08017/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 55 references that are in the map.
- First posted on arXiv 18 Aug 2021.

## Builds on (most cited first, 30 of 34)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Deep Image Prior](https://mlsanity.com/paper/1711.10925/) (2017)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images](https://mlsanity.com/paper/1804.01654/) (2018)
- [3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction](https://mlsanity.com/paper/1604.00449/) (2016)
- [Neural 3D Mesh Renderer](https://mlsanity.com/paper/1711.07566/) (2017)
- [Local Implicit Grid Representations for 3D Scenes](https://mlsanity.com/paper/2003.08981/) (2020)
- [Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion](https://mlsanity.com/paper/2003.01456/) (2020)
- [Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstruction](https://mlsanity.com/paper/2003.10983/) (2020)
- [Learning Category-Specific Mesh Reconstruction from Image Collections](https://mlsanity.com/paper/1803.07549/) (2018)
- [Texture Fields: Learning Texture Representations in Function Space](https://mlsanity.com/paper/1905.07259/) (2019)
- [Thingi10K: A Dataset of 10,000 3D-Printing Models](https://mlsanity.com/paper/1605.04797/) (2016)
- [Learning a Predictable and Generative Vector Representation for Objects](https://mlsanity.com/paper/1603.08637/) (2016)
- [Visual Object Networks: Image Generation with Disentangled 3D Representation](https://mlsanity.com/paper/1812.02725/) (2018)
- [Points2Surf: Learning Implicit Surfaces from Point Cloud Patches](https://mlsanity.com/paper/2007.10453/) (2020)
- ["Double-DIP": Unsupervised Image Decomposition via Coupled Deep-Image-Priors](https://mlsanity.com/paper/1812.00467/) (2018)
- [Neural Blind Deconvolution Using Deep Priors](https://mlsanity.com/paper/1908.02197/) (2019)
- [Deep Geometric Prior for Surface Reconstruction](https://mlsanity.com/paper/1811.10943/) (2018)
- [Pixel2Mesh++: Multi-View 3D Mesh Generation via Deformation](https://mlsanity.com/paper/1908.01491/) (2019)
- [Robust Watertight Manifold Surface Generation Method for ShapeNet Models](https://mlsanity.com/paper/1802.01698/) (2018)
- [Point2Mesh: A Self-Prior for Deformable Meshes](https://mlsanity.com/paper/2005.11084/) (2020)
- [Meshing Point Clouds with Predicted Intrinsic-Extrinsic Ratio Guidance](https://mlsanity.com/paper/2007.09267/) (2020)
- [SSRNet: Scalable 3D Surface Reconstruction Network](https://mlsanity.com/paper/1911.07401/) (2019)
- [Photorealistic Facial Texture Inference Using Deep Neural Networks](https://mlsanity.com/paper/1612.00523/) (2016)
- [Adversarial Texture Optimization from RGB-D Scans](https://mlsanity.com/paper/2003.08400/) (2020)
- [Image2Mesh: A Learning Framework for Single Image 3D Reconstruction](https://mlsanity.com/paper/1711.10669/) (2017)

## Built on by (most cited first, 13 of 13)

- [Controllable Mesh Generation Through Sparse Latent Point Diffusion Models](https://mlsanity.com/paper/2303.07938/) (2023)
- [3D Paintbrush: Local Stylization of 3D Shapes with Cascaded Score Distillation](https://mlsanity.com/paper/2311.09571/) (2023)
- [Ghost on the Shell: An Expressive Representation of General 3D Shapes](https://mlsanity.com/paper/2310.15168/) (2023)
- [SPARF: Large-Scale Learning of 3D Sparse Radiance Fields from Few Input Images](https://mlsanity.com/paper/2212.09100/) (2022)
- [What's the Situation with Intelligent Mesh Generation: A Survey and Perspectives](https://mlsanity.com/paper/2211.06009/) (2022)
- [SCoDA: Domain Adaptive Shape Completion for Real Scans](https://mlsanity.com/paper/2304.10179/) (2023)
- [Geometric Prior Based Deep Human Point Cloud Geometry Compression](https://mlsanity.com/paper/2305.01309/) (2023)
- [Spectrum AUC Difference (SAUCD): Human-aligned 3D Shape Evaluation](https://mlsanity.com/paper/2403.01619/) (2024)
- [LoRD: Local 4D Implicit Representation for High-Fidelity Dynamic Human Modeling](https://mlsanity.com/paper/2208.08622/) (2022)
- [NeuralMLS: Geometry-Aware Control Point Deformation](https://mlsanity.com/paper/2201.01873/) (2022)
- [PointDreamer: Zero-shot 3D Textured Mesh Reconstruction from Colored Point Cloud](https://mlsanity.com/paper/2406.15811/) (2024)
- [3D PixBrush: Image-Guided Local Texture Synthesis](https://mlsanity.com/paper/2507.03731/) (2025)
- [Learning Self-Prior for Mesh Inpainting Using Self-Supervised Graph Convolutional Networks](https://mlsanity.com/paper/2305.00635/) (2023)

## Abstract (arXiv)

We present a deep learning pipeline that leverages network self-prior to recover a full 3D model consisting of both a triangular mesh and a texture map from the colored 3D point cloud. Different from previous methods either exploiting 2D self-prior for image editing or 3D self-prior for pure surface reconstruction, we propose to exploit a novel hybrid 2D-3D self-prior in deep neural networks to significantly improve the geometry quality and produce a high-resolution texture map, which is typically missing from the output of commodity-level 3D scanners. In particular, we first generate an initial mesh using a 3D convolutional neural network with 3D self-prior, and then encode both 3D information and color information in the 2D UV atlas, which is further refined by 2D convolutional neural networks with the self-prior. In this way, both 2D and 3D self-priors are utilized for the mesh and texture recovery. Experiments show that, without the need of any additional training data, our method recovers the 3D textured mesh model of high quality from sparse input, and outperforms the state-of-the-art methods in terms of both the geometry and texture quality.