# SA-ConvONet: Sign-Agnostic Optimization of Convolutional Occupancy Networks

arXiv: https://arxiv.org/abs/2105.03582 · HTML: https://mlsanity.com/paper/2105.03582/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 50 references that are in the map.
- First posted on arXiv 8 May 2021.

## Builds on (most cited first, 30 of 39)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [Convolutional Occupancy Networks](https://mlsanity.com/paper/2003.04618/) (2020)
- [Implicit Geometric Regularization for Learning Shapes](https://mlsanity.com/paper/2002.10099/) (2020)
- [Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images](https://mlsanity.com/paper/1804.01654/) (2018)
- [3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction](https://mlsanity.com/paper/1604.00449/) (2016)
- [Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling](https://mlsanity.com/paper/1610.07584/) (2016)
- [Neural 3D Mesh Renderer](https://mlsanity.com/paper/1711.07566/) (2017)
- [SAL: Sign Agnostic Learning of Shapes from Raw Data](https://mlsanity.com/paper/1911.10414/) (2019)
- [Local Implicit Grid Representations for 3D Scenes](https://mlsanity.com/paper/2003.08981/) (2020)
- [Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion](https://mlsanity.com/paper/2003.01456/) (2020)
- [OctNet: Learning Deep 3D Representations at High Resolutions](https://mlsanity.com/paper/1611.05009/) (2016)
- [DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction](https://mlsanity.com/paper/1905.10711/) (2019)
- [Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstruction](https://mlsanity.com/paper/2003.10983/) (2020)
- [What Do Single-view 3D Reconstruction Networks Learn?](https://mlsanity.com/paper/1905.03678/) (2019)
- [Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs](https://mlsanity.com/paper/1703.09438/) (2017)
- [Points2Surf: Learning Implicit Surfaces from Point Cloud Patches](https://mlsanity.com/paper/2007.10453/) (2020)
- [SALD: Sign Agnostic Learning with Derivatives](https://mlsanity.com/paper/2006.05400/) (2020)
- [Deep Mesh Reconstruction from Single RGB Images via Topology Modification Networks](https://mlsanity.com/paper/1909.00321/) (2019)
- [PCPNET: Learning Local Shape Properties from Raw Point Clouds](https://mlsanity.com/paper/1710.04954/) (2017)
- [Hierarchical Surface Prediction for 3D Object Reconstruction](https://mlsanity.com/paper/1704.00710/) (2017)

## Built on by (most cited first, 30 of 59)

- [3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models](https://mlsanity.com/paper/2301.11445/) (2023)
- [CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets](https://mlsanity.com/paper/2406.13897/) (2024)
- [DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesis](https://mlsanity.com/paper/2303.14207/) (2023)
- [3DILG: Irregular Latent Grids for 3D Generative Modeling](https://mlsanity.com/paper/2205.13914/) (2022)
- [POCO: Point Convolution for Surface Reconstruction](https://mlsanity.com/paper/2201.01831/) (2022)
- [Neural Dual Contouring](https://mlsanity.com/paper/2202.01999/) (2022)
- [Reconstructing Surfaces for Sparse Point Clouds with On-Surface Priors](https://mlsanity.com/paper/2204.10603/) (2022)
- [Surface Reconstruction from Point Clouds by Learning Predictive Context Priors](https://mlsanity.com/paper/2204.11015/) (2022)
- [RGBD2: Generative Scene Synthesis via Incremental View Inpainting using RGBD Diffusion Models](https://mlsanity.com/paper/2212.05993/) (2022)
- [Surface Reconstruction from Point Clouds: A Survey and a Benchmark](https://mlsanity.com/paper/2205.02413/) (2022)
- [Towards Better Gradient Consistency for Neural Signed Distance Functions via Level Set Alignment](https://mlsanity.com/paper/2305.11601/) (2023)
- [Neural-Singular-Hessian: Implicit Neural Representation of Unoriented Point Clouds by Enforcing Singular Hessian](https://mlsanity.com/paper/2309.01793/) (2023)
- [Motion2VecSets: 4D Latent Vector Set Diffusion for Non-rigid Shape Reconstruction and Tracking](https://mlsanity.com/paper/2401.06614/) (2024)
- [Unsupervised Inference of Signed Distance Functions from Single Sparse Point Clouds without Learning Priors](https://mlsanity.com/paper/2303.14505/) (2023)
- [SE(3)-Equivariant Attention Networks for Shape Reconstruction in Function Space](https://mlsanity.com/paper/2204.02394/) (2022)
- [Neural Shape Deformation Priors](https://mlsanity.com/paper/2210.05616/) (2022)
- [Learning Signed Distance Functions from Noisy 3D Point Clouds via Noise to Noise Mapping](https://mlsanity.com/paper/2306.01405/) (2023)
- [PatchComplete: Learning Multi-Resolution Patch Priors for 3D Shape Completion on Unseen Categories](https://mlsanity.com/paper/2206.04916/) (2022)
- [Latent Partition Implicit with Surface Codes for 3D Representation](https://mlsanity.com/paper/2207.08631/) (2022)
- [ALTO: Alternating Latent Topologies for Implicit 3D Reconstruction](https://mlsanity.com/paper/2212.04096/) (2022)
- [RangeUDF: Semantic Surface Reconstruction from 3D Point Clouds](https://mlsanity.com/paper/2204.09138/) (2022)
- [Unsupervised Occupancy Learning from Sparse Point Cloud](https://mlsanity.com/paper/2404.02759/) (2024)
- [Fast Learning of Signed Distance Functions from Noisy Point Clouds via Noise to Noise Mapping](https://mlsanity.com/paper/2407.14225/) (2024)
- [Neural-IMLS: Self-supervised Implicit Moving Least-Squares Network for Surface Reconstruction](https://mlsanity.com/paper/2109.04398/) (2021)
- [Functional Diffusion](https://mlsanity.com/paper/2311.15435/) (2023)
- [Mixing-Denoising Generalizable Occupancy Networks](https://mlsanity.com/paper/2311.12125/) (2023)
- [NeurCADRecon: Neural Representation for Reconstructing CAD Surfaces by Enforcing Zero Gaussian Curvature](https://mlsanity.com/paper/2404.13420/) (2024)
- [Points2NeRF: Generating Neural Radiance Fields from 3D point cloud](https://mlsanity.com/paper/2206.01290/) (2022)
- [GridPull: Towards Scalability in Learning Implicit Representations from 3D Point Clouds](https://mlsanity.com/paper/2308.13175/) (2023)
- [Learning Neural Implicit through Volume Rendering with Attentive Depth Fusion Priors](https://mlsanity.com/paper/2310.11598/) (2023)

## Abstract (arXiv)

Surface reconstruction from point clouds is a fundamental problem in the computer vision and graphics community. Recent state-of-the-arts solve this problem by individually optimizing each local implicit field during inference. Without considering the geometric relationships between local fields, they typically require accurate normals to avoid the sign conflict problem in overlapped regions of local fields, which severely limits their applicability to raw scans where surface normals could be unavailable. Although SAL breaks this limitation via sign-agnostic learning, further works still need to explore how to extend this technique for local shape modeling. To this end, we propose to learn implicit surface reconstruction by sign-agnostic optimization of convolutional occupancy networks, to simultaneously achieve advanced scalability to large-scale scenes, generality to novel shapes, and applicability to raw scans in a unified framework. Concretely, we achieve this goal by a simple yet effective design, which further optimizes the pre-trained occupancy prediction networks with an unsigned cross-entropy loss during inference. The learning of occupancy fields is conditioned on convolutional features from an hourglass network architecture. Extensive experimental comparisons with previous state-of-the-arts on both object-level and scene-level datasets demonstrate the superior accuracy of our approach for surface reconstruction from un-orientated point clouds. The code is available at https://github.com/tangjiapeng/SA-ConvONet.