# Neural Fields as Learnable Kernels for 3D Reconstruction

arXiv: https://arxiv.org/abs/2111.13674 · HTML: https://mlsanity.com/paper/2111.13674/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- Builds on 60 of its 73 references that are in the map.
- First posted on arXiv 26 Nov 2021.

## Builds on (most cited first, 30 of 60)

- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [Convolutional Occupancy Networks](https://mlsanity.com/paper/2003.04618/) (2020)
- [Diffusion Probabilistic Models for 3D Point Cloud Generation](https://mlsanity.com/paper/2103.01458/) (2021)
- [Implicit Geometric Regularization for Learning Shapes](https://mlsanity.com/paper/2002.10099/) (2020)
- [3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction](https://mlsanity.com/paper/1604.00449/) (2016)
- [AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generation](https://mlsanity.com/paper/1802.05384/) (2018)
- [3D Shape Generation and Completion through Point-Voxel Diffusion](https://mlsanity.com/paper/2104.03670/) (2021)
- [Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis](https://mlsanity.com/paper/2111.04276/) (2021)
- [Deep Kernel Learning](https://mlsanity.com/paper/1511.02222/) (2015)
- [Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes](https://mlsanity.com/paper/2101.10994/) (2021)
- [SAL: Sign Agnostic Learning of Shapes from Raw Data](https://mlsanity.com/paper/1911.10414/) (2019)
- [Local Implicit Grid Representations for 3D Scenes](https://mlsanity.com/paper/2003.08981/) (2020)
- [Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion](https://mlsanity.com/paper/2003.01456/) (2020)
- [Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstruction](https://mlsanity.com/paper/2003.10983/) (2020)
- [Vector Neurons: A General Framework for SO(3)-Equivariant Networks](https://mlsanity.com/paper/2104.12229/) (2021)
- [What Do Single-view 3D Reconstruction Networks Learn?](https://mlsanity.com/paper/1905.03678/) (2019)
- [Learning Shape Templates with Structured Implicit Functions](https://mlsanity.com/paper/1904.06447/) (2019)
- [MetaSDF: Meta-learning Signed Distance Functions](https://mlsanity.com/paper/2006.09662/) (2020)
- [PU-Net: Point Cloud Upsampling Network](https://mlsanity.com/paper/1801.06761/) (2018)
- [BSP-Net: Generating Compact Meshes via Binary Space Partitioning](https://mlsanity.com/paper/1911.06971/) (2019)

## Built on by (most cited first, 30 of 58)

- [LION: Latent Point Diffusion Models for 3D Shape Generation](https://mlsanity.com/paper/2210.06978/) (2022)
- [3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models](https://mlsanity.com/paper/2301.11445/) (2023)
- [Neural Kernel Surface Reconstruction](https://mlsanity.com/paper/2305.19590/) (2023)
- [Dual Octree Graph Networks for Learning Adaptive Volumetric Shape Representations](https://mlsanity.com/paper/2205.02825/) (2022)
- [ALSO: Automotive Lidar Self-supervision by Occupancy estimation](https://mlsanity.com/paper/2212.05867/) (2022)
- [NeuRBF: A Neural Fields Representation with Adaptive Radial Basis Functions](https://mlsanity.com/paper/2309.15426/) (2023)
- [Surface Reconstruction from Point Clouds: A Survey and a Benchmark](https://mlsanity.com/paper/2205.02413/) (2022)
- [Point-Cloud Completion with Pretrained Text-to-image Diffusion Models](https://mlsanity.com/paper/2306.10533/) (2023)
- [SE(3)-Equivariant Attention Networks for Shape Reconstruction in Function Space](https://mlsanity.com/paper/2204.02394/) (2022)
- [Few 'Zero Level Set'-Shot Learning of Shape Signed Distance Functions in Feature Space](https://mlsanity.com/paper/2207.04161/) (2022)
- [ALTO: Alternating Latent Topologies for Implicit 3D Reconstruction](https://mlsanity.com/paper/2212.04096/) (2022)
- [Unsupervised Occupancy Learning from Sparse Point Cloud](https://mlsanity.com/paper/2404.02759/) (2024)
- [HyP-NeRF: Learning Improved NeRF Priors using a HyperNetwork](https://mlsanity.com/paper/2306.06093/) (2023)
- [NLOS-NeuS: Non-line-of-sight Neural Implicit Surface](https://mlsanity.com/paper/2303.12280/) (2023)
- [Mixing-Denoising Generalizable Occupancy Networks](https://mlsanity.com/paper/2311.12125/) (2023)
- [A Survey and Benchmark of Automatic Surface Reconstruction from Point Clouds](https://mlsanity.com/paper/2301.13656/) (2023)
- [PolyGNN: Polyhedron-based Graph Neural Network for 3D Building Reconstruction from Point Clouds](https://mlsanity.com/paper/2307.08636/) (2023)
- [Robustifying Generalizable Implicit Shape Networks with a Tunable Non-Parametric Model](https://mlsanity.com/paper/2311.12967/) (2023)
- [What's the Situation with Intelligent Mesh Generation: A Survey and Perspectives](https://mlsanity.com/paper/2211.06009/) (2022)
- [SCoDA: Domain Adaptive Shape Completion for Real Scans](https://mlsanity.com/paper/2304.10179/) (2023)
- [Zero-Shot Multi-Object Scene Completion](https://mlsanity.com/paper/2403.14628/) (2024)
- [Implicit Filtering for Learning Neural Signed Distance Functions from 3D Point Clouds](https://mlsanity.com/paper/2407.13342/) (2024)
- [Inferring Neural Signed Distance Functions by Overfitting on Single Noisy Point Clouds through Finetuning Data-Driven based Priors](https://mlsanity.com/paper/2410.19680/) (2024)
- [ROAD: Learning an Implicit Recursive Octree Auto-Decoder to Efficiently Encode 3D Shapes](https://mlsanity.com/paper/2212.06193/) (2022)
- [Registered and Segmented Deformable Object Reconstruction from a Single View Point Cloud](https://mlsanity.com/paper/2311.07357/) (2023)
- [Thera: Aliasing-Free Arbitrary-Scale Super-Resolution with Neural Heat Fields](https://mlsanity.com/paper/2311.17643/) (2023)
- [DiffCD: A Symmetric Differentiable Chamfer Distance for Neural Implicit Surface Fitting](https://mlsanity.com/paper/2407.17058/) (2024)
- [Neural Fields with Hard Constraints of Arbitrary Differential Order](https://mlsanity.com/paper/2306.08943/) (2023)
- [SweepNet: Unsupervised Learning Shape Abstraction via Neural Sweepers](https://mlsanity.com/paper/2407.06305/) (2024)
- [Few-Shot Unsupervised Implicit Neural Shape Representation Learning with Spatial Adversaries](https://mlsanity.com/paper/2408.15114/) (2024)

## Abstract (arXiv)

We present Neural Kernel Fields: a novel method for reconstructing implicit 3D shapes based on a learned kernel ridge regression. Our technique achieves state-of-the-art results when reconstructing 3D objects and large scenes from sparse oriented points, and can reconstruct shape categories outside the training set with almost no drop in accuracy. The core insight of our approach is that kernel methods are extremely effective for reconstructing shapes when the chosen kernel has an appropriate inductive bias. We thus factor the problem of shape reconstruction into two parts: (1) a backbone neural network which learns kernel parameters from data, and (2) a kernel ridge regression that fits the input points on-the-fly by solving a simple positive definite linear system using the learned kernel. As a result of this factorization, our reconstruction gains the benefits of data-driven methods under sparse point density while maintaining interpolatory behavior, which converges to the ground truth shape as input sampling density increases. Our experiments demonstrate a strong generalization capability to objects outside the train-set category and scanned scenes. Source code and pretrained models are available at https://nv-tlabs.github.io/nkf.