# FoV-NeRF: Foveated Neural Radiance Fields for Virtual Reality

arXiv: https://arxiv.org/abs/2103.16365 · HTML: https://mlsanity.com/paper/2103.16365/

## Key facts

- Cited by 79 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 59 references that are in the map.
- First posted on arXiv 30 Mar 2021.

## Builds on (most cited first, 27 of 27)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [Implicit Geometric Regularization for Learning Shapes](https://mlsanity.com/paper/2002.10099/) (2020)
- [Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines](https://mlsanity.com/paper/1905.00889/) (2019)
- [DeepVoxels: Learning Persistent 3D Feature Embeddings](https://mlsanity.com/paper/1812.01024/) (2018)
- [SAL: Sign Agnostic Learning of Shapes from Raw Data](https://mlsanity.com/paper/1911.10414/) (2019)
- [Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstruction](https://mlsanity.com/paper/2003.10983/) (2020)
- [NeX: Real-time View Synthesis with Neural Basis Expansion](https://mlsanity.com/paper/2103.05606/) (2021)
- [MetaSDF: Meta-learning Signed Distance Functions](https://mlsanity.com/paper/2006.09662/) (2020)
- [Texture Fields: Learning Texture Representations in Function Space](https://mlsanity.com/paper/1905.07259/) (2019)
- [AutoInt: Automatic Integration for Fast Neural Volume Rendering](https://mlsanity.com/paper/2012.01714/) (2020)
- [Learning-Based View Synthesis for Light Field Cameras](https://mlsanity.com/paper/1609.02974/) (2016)
- [Extreme View Synthesis](https://mlsanity.com/paper/1812.04777/) (2018)
- [PatchNets: Patch-Based Generalizable Deep Implicit 3D Shape Representations](https://mlsanity.com/paper/2008.01639/) (2020)
- [SDF-SRN: Learning Signed Distance 3D Object Reconstruction from Static Images](https://mlsanity.com/paper/2010.10505/) (2020)
- [MatryODShka: Real-time 6DoF Video View Synthesis using Multi-Sphere Images](https://mlsanity.com/paper/2008.06534/) (2020)
- [Deep Generative Modeling for Scene Synthesis via Hybrid Representations](https://mlsanity.com/paper/1808.02084/) (2018)
- [Transformation-Grounded Image Generation Network for Novel 3D View Synthesis](https://mlsanity.com/paper/1703.02921/) (2017)
- [Gaze-Contingent Ocular Parallax Rendering for Virtual Reality](https://mlsanity.com/paper/1906.09740/) (2019)
- [Deep Multi Depth Panoramas for View Synthesis](https://mlsanity.com/paper/2008.01815/) (2020)
- [Eccentricity Effects on Blur and Depth Perception](https://mlsanity.com/paper/1708.06034/) (2017)

## Built on by (most cited first, 30 of 79)

- [PGSR: Planar-based Gaussian Splatting for Efficient and High-Fidelity Surface Reconstruction](https://mlsanity.com/paper/2406.06521/) (2024)
- [WIRE: Wavelet Implicit Neural Representations](https://mlsanity.com/paper/2301.05187/) (2023)
- [GaussianPro: 3D Gaussian Splatting with Progressive Propagation](https://mlsanity.com/paper/2402.14650/) (2024)
- [VR-GS: A Physical Dynamics-Aware Interactive Gaussian Splatting System in Virtual Reality](https://mlsanity.com/paper/2401.16663/) (2024)
- [Depth-Regularized Optimization for 3D Gaussian Splatting in Few-Shot Images](https://mlsanity.com/paper/2311.13398/) (2023)
- [NeRFPlayer: A Streamable Dynamic Scene Representation with Decomposed Neural Radiance Fields](https://mlsanity.com/paper/2210.15947/) (2022)
- [NeRF-Art: Text-Driven Neural Radiance Fields Stylization](https://mlsanity.com/paper/2212.08070/) (2022)
- [Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fields](https://mlsanity.com/paper/2305.11588/) (2023)
- [SimpleNeRF: Regularizing Sparse Input Neural Radiance Fields with Simpler Solutions](https://mlsanity.com/paper/2309.03955/) (2023)
- [Improved Implicit Neural Representation with Fourier Reparameterized Training](https://mlsanity.com/paper/2401.07402/) (2024)
- [INCODE: Implicit Neural Conditioning with Prior Knowledge Embeddings](https://mlsanity.com/paper/2310.18846/) (2023)
- [VR-Splatting: Foveated Radiance Field Rendering via 3D Gaussian Splatting and Neural Points](https://mlsanity.com/paper/2410.17932/) (2024)
- [Learning Neural Duplex Radiance Fields for Real-Time View Synthesis](https://mlsanity.com/paper/2304.10537/) (2023)
- [SAGS: Structure-Aware 3D Gaussian Splatting](https://mlsanity.com/paper/2404.19149/) (2024)
- [VRSplat: Fast and Robust Gaussian Splatting for Virtual Reality](https://mlsanity.com/paper/2505.10144/) (2025)
- [Learning Transferable Features for Implicit Neural Representations](https://mlsanity.com/paper/2409.09566/) (2024)
- [TSGS: Improving Gaussian Splatting for Transparent Surface Reconstruction via Normal and De-lighting Priors](https://mlsanity.com/paper/2504.12799/) (2025)
- [Reconstructing 4D Spatial Intelligence: A Survey](https://mlsanity.com/paper/2507.21045/) (2025)
- [Eye-tracked Virtual Reality: A Comprehensive Survey on Methods and Privacy Challenges](https://mlsanity.com/paper/2305.14080/) (2023)
- [MuSHRoom: Multi-Sensor Hybrid Room Dataset for Joint 3D Reconstruction and Novel View Synthesis](https://mlsanity.com/paper/2311.02778/) (2023)
- [DeblurGS: Gaussian Splatting for Camera Motion Blur](https://mlsanity.com/paper/2404.11358/) (2024)
- [DisC-GS: Discontinuity-aware Gaussian Splatting](https://mlsanity.com/paper/2405.15196/) (2024)
- [Neural radiance fields in the industrial and robotics domain: applications, research opportunities and use cases](https://mlsanity.com/paper/2308.07118/) (2023)
- [Targeted Adversarial Attacks on Generalizable Neural Radiance Fields](https://mlsanity.com/paper/2310.03578/) (2023)
- [Semantics-Controlled Gaussian Splatting for Outdoor Scene Reconstruction and Rendering in Virtual Reality](https://mlsanity.com/paper/2409.15959/) (2024)
- [Bayesian NeRF: Quantifying Uncertainty with Volume Density for Neural Implicit Fields](https://mlsanity.com/paper/2404.06727/) (2024)
- [Frequency-Aware Density Control via Reparameterization for High-Quality Rendering of 3D Gaussian Splatting](https://mlsanity.com/paper/2503.07000/) (2025)
- [Improving Gaussian Splatting with Localized Points Management](https://mlsanity.com/paper/2406.04251/) (2024)
- [Inductive Gradient Adjustment For Spectral Bias In Implicit Neural Representations](https://mlsanity.com/paper/2410.13271/) (2024)
- [Turbo-GS: Accelerating 3D Gaussian Fitting for High-Quality Radiance Fields](https://mlsanity.com/paper/2412.13547/) (2024)

## Abstract (arXiv)

Virtual Reality (VR) is becoming ubiquitous with the rise of consumer displays and commercial VR platforms. Such displays require low latency and high quality rendering of synthetic imagery with reduced compute overheads. Recent advances in neural rendering showed promise of unlocking new possibilities in 3D computer graphics via image-based representations of virtual or physical environments. Specifically, the neural radiance fields (NeRF) demonstrated that photo-realistic quality and continuous view changes of 3D scenes can be achieved without loss of view-dependent effects. While NeRF can significantly benefit rendering for VR applications, it faces unique challenges posed by high field-of-view, high resolution, and stereoscopic/egocentric viewing, typically causing low quality and high latency of the rendered images. In VR, this not only harms the interaction experience but may also cause sickness. To tackle these problems toward six-degrees-of-freedom, egocentric, and stereo NeRF in VR, we present the first gaze-contingent 3D neural representation and view synthesis method. We incorporate the human psychophysics of visual- and stereo-acuity into an egocentric neural representation of 3D scenery. We then jointly optimize the latency/performance and visual quality while mutually bridging human perception and neural scene synthesis to achieve perceptually high-quality immersive interaction. We conducted both objective analysis and subjective studies to evaluate the effectiveness of our approach. We find that our method significantly reduces latency (up to 99% time reduction compared with NeRF) without loss of high-fidelity rendering (perceptually identical to full-resolution ground truth). The presented approach may serve as the first step toward future VR/AR systems that capture, teleport, and visualize remote environments in real-time.