# Instant-3D: Instant Neural Radiance Field Training Towards On-Device AR/VR 3D Reconstruction

arXiv: https://arxiv.org/abs/2304.12467 · HTML: https://mlsanity.com/paper/2304.12467/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Apr 2023.

## Built on by (most cited first, 30 of 51)

- [CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets](https://mlsanity.com/paper/2406.13897/) (2024)
- [L4GM: Large 4D Gaussian Reconstruction Model](https://mlsanity.com/paper/2406.10324/) (2024)
- [FlashGS: Efficient 3D Gaussian Splatting for Large-scale and High-resolution Rendering](https://mlsanity.com/paper/2408.07967/) (2024)
- [Animate3D: Animating Any 3D Model with Multi-view Video Diffusion](https://mlsanity.com/paper/2407.11398/) (2024)
- [DISTWAR: Fast Differentiable Rendering on Raster-based Rendering Pipelines](https://mlsanity.com/paper/2401.05345/) (2023)
- [EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild](https://mlsanity.com/paper/2411.14280/) (2024)
- [ZeroRF: Fast Sparse View 360deg Reconstruction with Zero Pretraining](https://mlsanity.com/paper/2312.09249/) (2023)
- [UniTEX: Universal High Fidelity Generative Texturing for 3D Shapes](https://mlsanity.com/paper/2505.23253/) (2025)
- [From Chaos to Clarity: 3DGS in the Dark](https://mlsanity.com/paper/2406.08300/) (2024)
- [PartRM: Modeling Part-Level Dynamics with Large Cross-State Reconstruction Model](https://mlsanity.com/paper/2503.19913/) (2025)
- [Reconstructing 4D Spatial Intelligence: A Survey](https://mlsanity.com/paper/2507.21045/) (2025)
- [UP-NeRF: Unconstrained Pose-Prior-Free Neural Radiance Fields](https://mlsanity.com/paper/2311.03784/) (2023)
- [Compress3D: a Compressed Latent Space for 3D Generation from a Single Image](https://mlsanity.com/paper/2403.13524/) (2024)
- [Auto-Regressively Generating Multi-View Consistent Images](https://mlsanity.com/paper/2506.18527/) (2025)
- [Finding Waldo: Towards Efficient Exploration of NeRF Scene Spaces](https://mlsanity.com/paper/2403.04508/) (2024)
- [FlexPainter: Flexible and Multi-View Consistent Texture Generation](https://mlsanity.com/paper/2506.02620/) (2025)
- [Object Isolated Attention for Consistent Story Visualization](https://mlsanity.com/paper/2503.23353/) (2025)
- [GauRast: Enhancing GPU Triangle Rasterizers to Accelerate 3D Gaussian Splatting](https://mlsanity.com/paper/2503.16681/) (2025)
- [GeoGS3D: Single-view 3D Reconstruction via Geometric-aware Diffusion Model and Gaussian Splatting](https://mlsanity.com/paper/2403.10242/) (2024)
- [Generative Human Geometry Distribution](https://mlsanity.com/paper/2503.01448/) (2025)
- [Neo: Real-Time On-Device 3D Gaussian Splatting with Reuse-and-Update Sorting Acceleration](https://mlsanity.com/paper/2511.12930/) (2025)
- [Aria-NeRF: Multimodal Egocentric View Synthesis](https://mlsanity.com/paper/2311.06455/) (2023)
- [Neural Rendering and Its Hardware Acceleration: A Review](https://mlsanity.com/paper/2402.00028/) (2024)
- [Jointly Generating Multi-view Consistent PBR Textures using Collaborative Control](https://mlsanity.com/paper/2410.06985/) (2024)
- [MASH: Masked Anchored SpHerical Distances for 3D Shape Representation and Generation](https://mlsanity.com/paper/2504.09149/) (2025)
- [ArtHOI: Taming Foundation Models for Monocular 4D Reconstruction of Hand-Articulated-Object Interactions](https://mlsanity.com/paper/2603.25791/) (2026)
- [VR-Pipe: Streamlining Hardware Graphics Pipeline for Volume Rendering](https://mlsanity.com/paper/2502.17078/) (2025)
- [Magic-Boost: Boost 3D Generation with Multi-View Conditioned Diffusion](https://mlsanity.com/paper/2404.06429/) (2024)
- [Towards Energy-Efficiency by Navigating the Trilemma of Energy, Latency, and Accuracy](https://mlsanity.com/paper/2409.04018/) (2024)
- [LucidFusion: Reconstructing 3D Gaussians with Arbitrary Unposed Images](https://mlsanity.com/paper/2410.15636/) (2024)

## Abstract (arXiv)

Neural Radiance Field (NeRF) based 3D reconstruction is highly desirable for immersive Augmented and Virtual Reality (AR/VR) applications, but achieving instant (i.e., < 5 seconds) on-device NeRF training remains a challenge. In this work, we first identify the inefficiency bottleneck: the need to interpolate NeRF embeddings up to 200,000 times from a 3D embedding grid during each training iteration. To alleviate this, we propose Instant-3D, an algorithm-hardware co-design acceleration framework that achieves instant on-device NeRF training. Our algorithm decomposes the embedding grid representation in terms of color and density, enabling computational redundancy to be squeezed out by adopting different (1) grid sizes and (2) update frequencies for the color and density branches. Our hardware accelerator further reduces the dominant memory accesses for embedding grid interpolation by (1) mapping multiple nearby points' memory read requests into one during the feed-forward process, (2) merging embedding grid updates from the same sliding time window during back-propagation, and (3) fusing different computation cores to support the different grid sizes needed by the color and density branches of Instant-3D algorithm. Extensive experiments validate the effectiveness of Instant-3D, achieving a large training time reduction of 41x - 248x while maintaining the same reconstruction quality. Excitingly, Instant-3D has enabled instant 3D reconstruction for AR/VR, requiring a reconstruction time of only 1.6 seconds per scene and meeting the AR/VR power consumption constraint of 1.9 W.