# High-Fidelity Eye Animatable Neural Radiance Fields for Human Face

arXiv: https://arxiv.org/abs/2308.00773 · HTML: https://mlsanity.com/paper/2308.00773/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 54 references that are in the map.
- First posted on arXiv 1 Aug 2023.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2007.02442/) (2020)
- [BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation](https://mlsanity.com/paper/1808.00897/) (2018)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video](https://mlsanity.com/paper/2201.04127/) (2022)
- [Learning an Animatable Detailed 3D Face Model from In-The-Wild Images](https://mlsanity.com/paper/2012.04012/) (2020)
- [Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction](https://mlsanity.com/paper/2012.03065/) (2020)
- [Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies](https://mlsanity.com/paper/2105.02872/) (2021)
- [AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis](https://mlsanity.com/paper/2103.11078/) (2021)
- [Learning from Simulated and Unsupervised Images through Adversarial Training](https://mlsanity.com/paper/1612.07828/) (2016)
- [HeadNeRF: A Real-time NeRF-based Parametric Head Model](https://mlsanity.com/paper/2112.05637/) (2021)
- [Gaze360: Physically Unconstrained Gaze Estimation in the Wild](https://mlsanity.com/paper/1910.10088/) (2019)
- [ETH-XGaze: A Large Scale Dataset for Gaze Estimation under Extreme Head Pose and Gaze Variation](https://mlsanity.com/paper/2007.15837/) (2020)
- [GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesis](https://mlsanity.com/paper/2301.13430/) (2023)
- [It's Written All Over Your Face: Full-Face Appearance-Based Gaze Estimation](https://mlsanity.com/paper/1611.08860/) (2016)
- [MPIIGaze: Real-World Dataset and Deep Appearance-Based Gaze Estimation](https://mlsanity.com/paper/1711.09017/) (2017)
- [Appearance-based Gaze Estimation With Deep Learning: A Review and Benchmark](https://mlsanity.com/paper/2104.12668/) (2021)
- [Few-Shot Adaptive Gaze Estimation](https://mlsanity.com/paper/1905.01941/) (2019)
- [Gaze Estimation using Transformer](https://mlsanity.com/paper/2105.14424/) (2021)
- [Face Aging With Conditional Generative Adversarial Networks](https://mlsanity.com/paper/1702.01983/) (2017)
- [Rendering of Eyes for Eye-Shape Registration and Gaze Estimation](https://mlsanity.com/paper/1505.05916/) (2015)
- [HS-Pose: Hybrid Scope Feature Extraction for Category-level Object Pose Estimation](https://mlsanity.com/paper/2303.15743/) (2023)
- [DAE-Talker: High Fidelity Speech-Driven Talking Face Generation with Diffusion Autoencoder](https://mlsanity.com/paper/2303.17550/) (2023)
- [Improving Few-Shot User-Specific Gaze Adaptation via Gaze Redirection Synthesis](https://mlsanity.com/paper/1904.10638/) (2019)
- [Recursive-NeRF: An Efficient and Dynamically Growing NeRF](https://mlsanity.com/paper/2105.09103/) (2021)
- [Self-Learning Transformations for Improving Gaze and Head Redirection](https://mlsanity.com/paper/2010.12307/) (2020)

## Built on by (most cited first, 6 of 6)

- [DVGaze: Dual-View Gaze Estimation](https://mlsanity.com/paper/2308.10310/) (2023)
- [NL2Contact: Natural Language Guided 3D Hand-Object Contact Modeling with Diffusion Model](https://mlsanity.com/paper/2407.12727/) (2024)
- [GazeGaussian: High-Fidelity Gaze Redirection with 3D Gaussian Splatting](https://mlsanity.com/paper/2411.12981/) (2024)
- [UniGaze: Towards Universal Gaze Estimation via Large-scale Pre-Training](https://mlsanity.com/paper/2502.02307/) (2025)
- [Roll Your Eyes: Gaze Redirection via Explicit 3D Eyeball Rotation](https://mlsanity.com/paper/2508.06136/) (2025)
- [GazePrior: Zero-Shot AR/VR Eye Tracking via Learned 3D Gaze Reconstruction](https://mlsanity.com/paper/2605.22359/) (2026)

## Abstract (arXiv)

Face rendering using neural radiance fields (NeRF) is a rapidly developing research area in computer vision. While recent methods primarily focus on controlling facial attributes such as identity and expression, they often overlook the crucial aspect of modeling eyeball rotation, which holds importance for various downstream tasks. In this paper, we aim to learn a face NeRF model that is sensitive to eye movements from multi-view images. We address two key challenges in eye-aware face NeRF learning: how to effectively capture eyeball rotation for training and how to construct a manifold for representing eyeball rotation. To accomplish this, we first fit FLAME, a well-established parametric face model, to the multi-view images considering multi-view consistency. Subsequently, we introduce a new Dynamic Eye-aware NeRF (DeNeRF). DeNeRF transforms 3D points from different views into a canonical space to learn a unified face NeRF model. We design an eye deformation field for the transformation, including rigid transformation, e.g., eyeball rotation, and non-rigid transformation. Through experiments conducted on the ETH-XGaze dataset, we demonstrate that our model is capable of generating high-fidelity images with accurate eyeball rotation and non-rigid periocular deformation, even under novel viewing angles. Furthermore, we show that utilizing the rendered images can effectively enhance gaze estimation performance.