# Fast Training of Neural Lumigraph Representations using Meta Learning

arXiv: https://arxiv.org/abs/2106.14942 · HTML: https://mlsanity.com/paper/2106.14942/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 65 of its 102 references that are in the map.
- First posted on arXiv 28 Jun 2021.

## Builds on (most cited first, 30 of 65)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [Volume Rendering of Neural Implicit Surfaces](https://mlsanity.com/paper/2106.12052/) (2021)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [PlenOctrees for Real-time Rendering of Neural Radiance Fields](https://mlsanity.com/paper/2103.14024/) (2021)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields](https://mlsanity.com/paper/2011.12100/) (2020)
- [Convolutional Occupancy Networks](https://mlsanity.com/paper/2003.04618/) (2020)
- [Implicit Geometric Regularization for Learning Shapes](https://mlsanity.com/paper/2002.10099/) (2020)
- [pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2012.00926/) (2020)
- [IBRNet: Learning Multi-View Image-Based Rendering](https://mlsanity.com/paper/2102.13090/) (2021)
- [Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision](https://mlsanity.com/paper/1912.07372/) (2019)
- [GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2007.02442/) (2020)
- [KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs](https://mlsanity.com/paper/2103.13744/) (2021)
- [Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines](https://mlsanity.com/paper/1905.00889/) (2019)

## Built on by (most cited first, 29 of 29)

- [RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs](https://mlsanity.com/paper/2112.00724/) (2021)
- [Neural Fields in Visual Computing and Beyond](https://mlsanity.com/paper/2111.11426/) (2021)
- [Advances in Neural Rendering](https://mlsanity.com/paper/2111.05849/) (2021)
- [HeadNeRF: A Real-time NeRF-based Parametric Head Model](https://mlsanity.com/paper/2112.05637/) (2021)
- [BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis](https://mlsanity.com/paper/2302.14859/) (2023)
- [Neural Rays for Occlusion-aware Image-based Rendering](https://mlsanity.com/paper/2107.13421/) (2021)
- [Improving neural implicit surfaces geometry with patch warping](https://mlsanity.com/paper/2112.09648/) (2021)
- [Grid-guided Neural Radiance Fields for Large Urban Scenes](https://mlsanity.com/paper/2303.14001/) (2023)
- [Learning Neural Volumetric Representations of Dynamic Humans in Minutes](https://mlsanity.com/paper/2302.12237/) (2023)
- [Unified Implicit Neural Stylization](https://mlsanity.com/paper/2204.01943/) (2022)
- [Im4D: High-Fidelity and Real-Time Novel View Synthesis for Dynamic Scenes](https://mlsanity.com/paper/2310.08585/) (2023)
- [MixNeRF: Modeling a Ray with Mixture Density for Novel View Synthesis from Sparse Inputs](https://mlsanity.com/paper/2302.08788/) (2023)
- [NeROIC: Neural Rendering of Objects from Online Image Collections](https://mlsanity.com/paper/2201.02533/) (2022)
- [3D-Aware Video Generation](https://mlsanity.com/paper/2206.14797/) (2022)
- [VaxNeRF: Revisiting the Classic for Voxel-Accelerated Neural Radiance Field](https://mlsanity.com/paper/2111.13112/) (2021)
- [TINC: Tree-structured Implicit Neural Compression](https://mlsanity.com/paper/2211.06689/) (2022)
- [SCI: A Spectrum Concentrated Implicit Neural Compression for Biomedical Data](https://mlsanity.com/paper/2209.15180/) (2022)
- [Learning to Learn and Sample BRDFs](https://mlsanity.com/paper/2210.03510/) (2022)
- [Instant Continual Learning of Neural Radiance Fields](https://mlsanity.com/paper/2309.01811/) (2023)
- [LiveNVS: Neural View Synthesis on Live RGB-D Streams](https://mlsanity.com/paper/2311.16668/) (2023)
- [CodecNeRF: Toward Fast Encoding and Decoding, Compact, and High-quality Novel-view Synthesis](https://mlsanity.com/paper/2404.04913/) (2024)
- [VIINTER: View Interpolation with Implicit Neural Representations of Images](https://mlsanity.com/paper/2211.00722/) (2022)
- [Scalable and High-Quality Neural Implicit Representation for 3D Reconstruction](https://mlsanity.com/paper/2501.08577/) (2025)
- [AlteredAvatar: Stylizing Dynamic 3D Avatars with Fast Style Adaptation](https://mlsanity.com/paper/2305.19245/) (2023)
- [AltNeRF: Learning Robust Neural Radiance Field via Alternating Depth-Pose Optimization](https://mlsanity.com/paper/2308.10001/) (2023)
- [Deep representation learning: Fundamentals, Perspectives, Applications, and Open Challenges](https://mlsanity.com/paper/2211.14732/) (2022)
- [MetaDIP: Accelerating Deep Image Prior with Meta Learning](https://mlsanity.com/paper/2209.08452/) (2022)
- [Edge Preserving Implicit Surface Representation of Point Clouds](https://mlsanity.com/paper/2301.04860/) (2023)
- [An Exploration of Neural Radiance Field Scene Reconstruction: Synthetic, Real-world and Dynamic Scenes](https://mlsanity.com/paper/2210.12268/) (2022)

## Abstract (arXiv)

Novel view synthesis is a long-standing problem in machine learning and computer vision. Significant progress has recently been made in developing neural scene representations and rendering techniques that synthesize photorealistic images from arbitrary views. These representations, however, are extremely slow to train and often also slow to render. Inspired by neural variants of image-based rendering, we develop a new neural rendering approach with the goal of quickly learning a high-quality representation which can also be rendered in real-time. Our approach, MetaNLR++, accomplishes this by using a unique combination of a neural shape representation and 2D CNN-based image feature extraction, aggregation, and re-projection. To push representation convergence times down to minutes, we leverage meta learning to learn neural shape and image feature priors which accelerate training. The optimized shape and image features can then be extracted using traditional graphics techniques and rendered in real time. We show that MetaNLR++ achieves similar or better novel view synthesis results in a fraction of the time that competing methods require.