# DiffS-NOCS: 3D Point Cloud Reconstruction through Coloring Sketches to NOCS Maps Using Diffusion Models

arXiv: https://arxiv.org/abs/2506.12835 · HTML: https://mlsanity.com/paper/2506.12835/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 27 of its 34 references that are in the map.
- First posted on arXiv 15 Jun 2025.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [Vector Quantized Diffusion Model for Text-to-Image Synthesis](https://mlsanity.com/paper/2111.14822/) (2021)
- [Diffusion Probabilistic Models for 3D Point Cloud Generation](https://mlsanity.com/paper/2103.01458/) (2021)
- [3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction](https://mlsanity.com/paper/1604.00449/) (2016)
- [3D Shape Generation and Completion through Point-Voxel Diffusion](https://mlsanity.com/paper/2104.03670/) (2021)
- [Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation](https://mlsanity.com/paper/1901.02970/) (2019)
- [LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models](https://mlsanity.com/paper/2309.15103/) (2023)
- [Photorealistic Video Generation with Diffusion Models](https://mlsanity.com/paper/2312.06662/) (2023)
- [Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings](https://mlsanity.com/paper/1803.08495/) (2018)
- [Locally Attentional SDF Diffusion for Controllable 3D Shape Generation](https://mlsanity.com/paper/2305.04461/) (2023)
- [MarrNet: 3D Shape Reconstruction via 2.5D Sketches](https://mlsanity.com/paper/1711.03129/) (2017)
- [Sketch2Model: View-Aware 3D Modeling from Single Free-Hand Sketches](https://mlsanity.com/paper/2105.06663/) (2021)
- [Sketch2Mesh: Reconstructing and Editing 3D Shapes from Sketches](https://mlsanity.com/paper/2104.00482/) (2021)
- [VolumeFusion: Deep Depth Fusion for 3D Scene Reconstruction](https://mlsanity.com/paper/2108.08623/) (2021)
- [Sketch Your Own GAN](https://mlsanity.com/paper/2108.02774/) (2021)
- [Multiview Aggregation for Learning Category-Specific Shape Reconstruction](https://mlsanity.com/paper/1907.01085/) (2019)

## Abstract (arXiv)

Reconstructing a 3D point cloud from a given conditional sketch is challenging. Existing methods often work directly in 3D space, but domain variability and difficulty in reconstructing accurate 3D structures from 2D sketches remain significant obstacles. Moreover, ideal models should also accept prompts for control, in addition with the sparse sketch, posing challenges in multi-modal fusion. We propose DiffS-NOCS (Diffusion-based Sketch-to-NOCS Map), which leverages ControlNet with a modified multi-view decoder to generate NOCS maps with embedded 3D structure and position information in 2D space from sketches. The 3D point cloud is reconstructed by combining multiple NOCS maps from different views. To enhance sketch understanding, we integrate a viewpoint encoder for extracting viewpoint features. Additionally, we design a feature-level multi-view aggregation network as the denoising module, facilitating cross-view information exchange and improving 3D consistency in NOCS map generation. Experiments on ShapeNet demonstrate that DiffS-NOCS achieves controllable and fine-grained point cloud reconstruction aligned with sketches.