# RT-NeRF: Real-Time On-Device Neural Radiance Fields Towards Immersive AR/VR Rendering

arXiv: https://arxiv.org/abs/2212.01120 · HTML: https://mlsanity.com/paper/2212.01120/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Dec 2022.

## Built on by (most cited first, 30 of 36)

- [VR-GS: A Physical Dynamics-Aware Interactive Gaussian Splatting System in Virtual Reality](https://mlsanity.com/paper/2401.16663/) (2024)
- [VFusion3D: Learning Scalable 3D Generative Models from Video Diffusion Models](https://mlsanity.com/paper/2403.12034/) (2024)
- [DISTWAR: Fast Differentiable Rendering on Raster-based Rendering Pipelines](https://mlsanity.com/paper/2401.05345/) (2023)
- [Baking Gaussian Splatting into Diffusion Denoiser for Fast and Scalable Single-stage Image-to-3D Generation and Reconstruction](https://mlsanity.com/paper/2411.14384/) (2024)
- [LangSurf: Language-Embedded Surface Gaussians for 3D Scene Understanding](https://mlsanity.com/paper/2412.17635/) (2024)
- [SlimmeRF: Slimmable Radiance Fields](https://mlsanity.com/paper/2312.10034/) (2023)
- [From Chaos to Clarity: 3DGS in the Dark](https://mlsanity.com/paper/2406.08300/) (2024)
- [Flex3D: Feed-Forward 3D Generation with Flexible Reconstruction Model and Input View Curation](https://mlsanity.com/paper/2410.00890/) (2024)
- [Semantics-Controlled Gaussian Splatting for Outdoor Scene Reconstruction and Rendering in Virtual Reality](https://mlsanity.com/paper/2409.15959/) (2024)
- [Gen-NeRF: Efficient and Generalizable Neural Radiance Fields via Algorithm-Hardware Co-Design](https://mlsanity.com/paper/2304.11842/) (2023)
- [Strata-NeRF : Neural Radiance Fields for Stratified Scenes](https://mlsanity.com/paper/2308.10337/) (2023)
- [Multi-Layer Gaussian Splatting for Immersive Anatomy Visualization](https://mlsanity.com/paper/2410.16978/) (2024)
- [GS-Cache: A GS-Cache Inference Framework for Large-scale Gaussian Splatting Models](https://mlsanity.com/paper/2502.14938/) (2025)
- [LiveNVS: Neural View Synthesis on Live RGB-D Streams](https://mlsanity.com/paper/2311.16668/) (2023)
- [A Survey on 3D Reconstruction Techniques in Plant Phenotyping: From Classical Methods to Neural Radiance Fields (NeRF), 3D Gaussian Splatting (3DGS), and Beyond](https://mlsanity.com/paper/2505.00737/) (2025)
- [ClusteringSDF: Self-Organized Neural Implicit Surfaces for 3D Decomposition](https://mlsanity.com/paper/2403.14619/) (2024)
- [Neo: Real-Time On-Device 3D Gaussian Splatting with Reuse-and-Update Sorting Acceleration](https://mlsanity.com/paper/2511.12930/) (2025)
- [MF-NeRF: Memory Efficient NeRF with Mixed-Feature Hash Table](https://mlsanity.com/paper/2304.12587/) (2023)
- [MultiPlaneNeRF: Neural Radiance Field with Non-Trainable Representation](https://mlsanity.com/paper/2305.10579/) (2023)
- [MARF: The Medial Atom Ray Field Object Representation](https://mlsanity.com/paper/2307.00037/) (2023)
- [FG-NeRF: Flow-GAN based Probabilistic Neural Radiance Field for Independence-Assumption-Free Uncertainty Estimation](https://mlsanity.com/paper/2309.16364/) (2023)
- [REFRAME: Reflective Surface Real-Time Rendering for Mobile Devices](https://mlsanity.com/paper/2403.16481/) (2024)
- [Uni-Render: A Unified Accelerator for Real-Time Rendering Across Diverse Neural Renderers](https://mlsanity.com/paper/2503.23644/) (2025)
- [FusedRF: Fusing Multiple Radiance Fields](https://mlsanity.com/paper/2306.04180/) (2023)
- [Voxel-Mesh Hybrid Representation for Real-Time View Synthesis](https://mlsanity.com/paper/2403.06505/) (2024)
- [SpNeRF: Memory Efficient Sparse Volumetric Neural Rendering Accelerator for Edge Devices](https://mlsanity.com/paper/2505.08191/) (2025)
- [Radiant Foam Rendering on a Graph Processor](https://mlsanity.com/paper/2601.04382/) (2026)
- [VersaQ-3D: Architecture Support for Visual Geometry Grounded Transformers via Versatile Quantization](https://mlsanity.com/paper/2601.20317/) (2026)
- [Factorized Multi-Resolution HashGrid for Efficient Neural Radiance Fields: Execution on Edge-Devices](https://mlsanity.com/paper/2604.02836/) (2026)
- [Splaxel: Efficient Distributed Training of 3D Gaussian Splatting for Large-scale Scene Reconstruction via Pixel-level Communication](https://mlsanity.com/paper/2606.18588/) (2026)

## Abstract (arXiv)

Neural Radiance Field (NeRF) based rendering has attracted growing attention thanks to its state-of-the-art (SOTA) rendering quality and wide applications in Augmented and Virtual Reality (AR/VR). However, immersive real-time (> 30 FPS) NeRF based rendering enabled interactions are still limited due to the low achievable throughput on AR/VR devices. To this end, we first profile SOTA efficient NeRF algorithms on commercial devices and identify two primary causes of the aforementioned inefficiency: (1) the uniform point sampling and (2) the dense accesses and computations of the required embeddings in NeRF. Furthermore, we propose RT-NeRF, which to the best of our knowledge is the first algorithm-hardware co-design acceleration of NeRF. Specifically, on the algorithm level, RT-NeRF integrates an efficient rendering pipeline for largely alleviating the inefficiency due to the commonly adopted uniform point sampling method in NeRF by directly computing the geometry of pre-existing points. Additionally, RT-NeRF leverages a coarse-grained view-dependent computing ordering scheme for eliminating the (unnecessary) processing of invisible points. On the hardware level, our proposed RT-NeRF accelerator (1) adopts a hybrid encoding scheme to adaptively switch between a bitmap- or coordinate-based sparsity encoding format for NeRF's sparse embeddings, aiming to maximize the storage savings and thus reduce the required DRAM accesses while supporting efficient NeRF decoding; and (2) integrates both a dual-purpose bi-direction adder & search tree and a high-density sparse search unit to coordinate the two aforementioned encoding formats. Extensive experiments on eight datasets consistently validate the effectiveness of RT-NeRF, achieving a large throughput improvement (e.g., 9.7x - 3,201x) while maintaining the rendering quality as compared with SOTA efficient NeRF solutions.