# A3FR: Agile 3D Gaussian Splatting with Incremental Gaze Tracked Foveated Rendering in Virtual Reality

arXiv: https://arxiv.org/abs/2507.04147 · HTML: https://mlsanity.com/paper/2507.04147/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 76 references that are in the map.
- First posted on arXiv 5 Jul 2025.

## Builds on (most cited first, 20 of 20)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning](https://mlsanity.com/paper/2406.08858/) (2024)
- [Appearance-Based Gaze Estimation in the Wild](https://mlsanity.com/paper/1504.02863/) (2015)
- [MPIIGaze: Real-World Dataset and Deep Appearance-Based Gaze Estimation](https://mlsanity.com/paper/1711.09017/) (2017)
- [NeRF-DS: Neural Radiance Fields for Dynamic Specular Objects](https://mlsanity.com/paper/2303.14435/) (2023)
- [Event Based, Near Eye Gaze Tracking Beyond 10,000Hz](https://mlsanity.com/paper/2004.03577/) (2020)
- [RITnet: Real-time Semantic Segmentation of the Eye for Gaze Tracking](https://mlsanity.com/paper/1910.00694/) (2019)
- [Evaluation of Appearance-Based Methods and Implications for Gaze-Based Applications](https://mlsanity.com/paper/1901.10906/) (2019)
- [MetaSapiens: Real-Time Neural Rendering with Efficiency-Aware Pruning and Accelerated Foveated Rendering](https://mlsanity.com/paper/2407.00435/) (2024)
- [Real-Time Gaze Tracking with Event-Driven Eye Segmentation](https://mlsanity.com/paper/2201.07367/) (2022)
- [OpenEDS2020: Open Eyes Dataset](https://mlsanity.com/paper/2005.03876/) (2020)
- [Co-designing a Sub-millisecond Latency Event-based Eye Tracking System with Submanifold Sparse CNN](https://mlsanity.com/paper/2404.14279/) (2024)
- [AMReX: Block-Structured Adaptive Mesh Refinement for Multiphysics Applications](https://mlsanity.com/paper/2009.12009/) (2020)
- [One Eye is All You Need: Lightweight Ensembles for Gaze Estimation with Single Encoders](https://mlsanity.com/paper/2211.11936/) (2022)
- [Saccade-Contingent Rendering](https://mlsanity.com/paper/2401.16536/) (2024)
- [BOXR: Body and head motion Optimization framework for eXtended Reality](https://mlsanity.com/paper/2410.13084/) (2024)
- [Neural Architecture Search of Hybrid Models for NPU-CIM Heterogeneous AR/VR Devices](https://mlsanity.com/paper/2410.08326/) (2024)
- [FovealNet: Advancing AI-Driven Gaze Tracking Solutions for Optimized Foveated Rendering System Performance in Virtual Reality](https://mlsanity.com/paper/2412.10456/) (2024)

## Abstract (arXiv)

Virtual reality (VR) significantly transforms immersive digital interfaces, greatly enhancing education, professional practices, and entertainment by increasing user engagement and opening up new possibilities in various industries. Among its numerous applications, image rendering is crucial. Nevertheless, rendering methodologies like 3D Gaussian Splatting impose high computational demands, driven predominantly by user expectations for superior visual quality. This results in notable processing delays for real-time image rendering, which greatly affects the user experience. Additionally, VR devices such as head-mounted displays (HMDs) are intricately linked to human visual behavior, leveraging knowledge from perception and cognition to improve user experience. These insights have spurred the development of foveated rendering, a technique that dynamically adjusts rendering resolution based on the user's gaze direction. The resultant solution, known as gaze-tracked foveated rendering, significantly reduces the computational burden of the rendering process. Although gaze-tracked foveated rendering can reduce rendering costs, the computational overhead of the gaze tracking process itself can sometimes outweigh the rendering savings, leading to increased processing latency. To address this issue, we propose an efficient rendering framework called~textitA3FR, designed to minimize the latency of gaze-tracked foveated rendering via the parallelization of gaze tracking and foveated rendering processes. For the rendering algorithm, we utilize 3D Gaussian Splatting, a state-of-the-art neural rendering technique. Evaluation results demonstrate that A3FR can reduce end-to-end rendering latency by up to 2× while maintaining visual quality.