# Efficient Region-Aware Neural Radiance Fields for High-Fidelity Talking Portrait Synthesis

arXiv: https://arxiv.org/abs/2307.09323 · HTML: https://mlsanity.com/paper/2307.09323/

## Key facts

- Cited by 72 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 58 references that are in the map.
- First posted on arXiv 18 Jul 2023.

## Builds on (most cited first, 30 of 38)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [TensoRF: Tensorial Radiance Fields](https://mlsanity.com/paper/2203.09517/) (2022)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021)
- [K-Planes: Explicit Radiance Fields in Space, Time, and Appearance](https://mlsanity.com/paper/2301.10241/) (2023)
- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [HexPlane: A Fast Representation for Dynamic Scenes](https://mlsanity.com/paper/2301.09632/) (2023)
- [Deep Speech: Scaling up end-to-end speech recognition](https://mlsanity.com/paper/1412.5567/) (2014)
- [Fast Dynamic Radiance Fields with Time-Aware Neural Voxels](https://mlsanity.com/paper/2205.15285/) (2022)
- [Multi-attentional Deepfake Detection](https://mlsanity.com/paper/2103.02406/) (2021)
- [AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis](https://mlsanity.com/paper/2103.11078/) (2021)
- [Detecting Deepfakes with Self-Blended Images](https://mlsanity.com/paper/2204.08376/) (2022)
- [MakeItTalk: Speaker-Aware Talking-Head Animation](https://mlsanity.com/paper/2004.12992/) (2020)
- [Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation](https://mlsanity.com/paper/2104.11116/) (2021)
- [Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Loss](https://mlsanity.com/paper/1905.03820/) (2019)
- [Neural Voice Puppetry: Audio-driven Facial Reenactment](https://mlsanity.com/paper/1912.05566/) (2019)
- [X2Face: A network for controlling face generation by using images, audio, and pose codes](https://mlsanity.com/paper/1807.10550/) (2018)
- [Self-supervised Learning of Adversarial Example: Towards Good Generalizations for Deepfake Detection](https://mlsanity.com/paper/2203.12208/) (2022)
- [Lip Movements Generation at a Glance](https://mlsanity.com/paper/1803.10404/) (2018)
- [NeRFPlayer: A Streamable Dynamic Scene Representation with Decomposed Neural Radiance Fields](https://mlsanity.com/paper/2210.15947/) (2022)
- [Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation](https://mlsanity.com/paper/2301.03396/) (2023)
- [DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animation](https://mlsanity.com/paper/2301.03786/) (2023)
- [GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesis](https://mlsanity.com/paper/2301.13430/) (2023)
- [Mixed Neural Voxels for Fast Multi-view Video Synthesis](https://mlsanity.com/paper/2212.00190/) (2022)
- [Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation](https://mlsanity.com/paper/2201.07786/) (2022)
- [Beyond Self-attention: External Attention using Two Linear Layers for Visual Tasks](https://mlsanity.com/paper/2105.02358/) (2021)
- [Learning Dynamic Facial Radiance Fields for Few-Shot Talking Head Synthesis](https://mlsanity.com/paper/2207.11770/) (2022)

## Built on by (most cited first, 30 of 72)

- [SyncTalk: The Devil is in the Synchronization for Talking Head Synthesis](https://mlsanity.com/paper/2311.17590/) (2023)
- [TalkingGaussian: Structure-Persistent 3D Talking Head Synthesis via Gaussian Splatting](https://mlsanity.com/paper/2404.15264/) (2024)
- [GaussianHead: High-fidelity Head Avatars with Learnable Gaussian Derivation](https://mlsanity.com/paper/2312.01632/) (2023)
- [Ditto: Motion-Space Diffusion for Controllable Realtime Talking Head Synthesis](https://mlsanity.com/paper/2411.19509/) (2024)
- [GaussianTalker: Speaker-specific Talking Head Synthesis via 3D Gaussian Splatting](https://mlsanity.com/paper/2404.14037/) (2024)
- [InsTaG: Learning Personalized 3D Talking Head from Few-Second Video](https://mlsanity.com/paper/2502.20387/) (2025)
- [AlignMiF: Geometry-Aligned Multimodal Implicit Field for LiDAR-Camera Joint Synthesis](https://mlsanity.com/paper/2402.17483/) (2024)
- [GaussianTalker: Real-Time High-Fidelity Talking Head Synthesis with Audio-Driven 3D Gaussian Splatting](https://mlsanity.com/paper/2404.16012/) (2024)
- [Tri2-plane: Thinking Head Avatar via Feature Pyramid](https://mlsanity.com/paper/2401.09386/) (2024)
- [EmoTalk3D: High-Fidelity Free-View Synthesis of Emotional 3D Talking Head](https://mlsanity.com/paper/2408.00297/) (2024)
- [MimicTalk: Mimicking a personalized and expressive 3D talking face in minutes](https://mlsanity.com/paper/2410.06734/) (2024)
- [MagicInfinite: Generating Infinite Talking Videos with Your Words and Voice](https://mlsanity.com/paper/2503.05978/) (2025)
- [PersonaTalk: Bring Attention to Your Persona in Visual Dubbing](https://mlsanity.com/paper/2409.05379/) (2024)
- [MegActor-Sigma: Unlocking Flexible Mixed-Modal Control in Portrait Animation with Diffusion Transformer](https://mlsanity.com/paper/2408.14975/) (2024)
- [Learning Dense Correspondence for NeRF-Based Face Reenactment](https://mlsanity.com/paper/2312.10422/) (2023)
- [Learning Dynamic Tetrahedra for High-Quality Talking Head Synthesis](https://mlsanity.com/paper/2402.17364/) (2024)
- [Neural Radiance Field-based Visual Rendering: A Comprehensive Review](https://mlsanity.com/paper/2404.00714/) (2024)
- [GSTalker: Real-time Audio-Driven Talking Face Generation via Deformable Gaussian Splatting](https://mlsanity.com/paper/2404.19040/) (2024)
- [BIGS: Bimanual Category-agnostic Interaction Reconstruction from Monocular Videos via 3D Gaussian Splatting](https://mlsanity.com/paper/2504.09097/) (2025)
- [SyncTalk++: High-Fidelity and Efficient Synchronized Talking Heads Synthesis Using Gaussian Splatting](https://mlsanity.com/paper/2506.14742/) (2025)
- [CorrTalk: Correlation Between Hierarchical Speech and Facial Activity Variances for 3D Animation](https://mlsanity.com/paper/2310.11295/) (2023)
- [NeRFFaceSpeech: One-shot Audio-driven 3D Talking Head Synthesis via Generative Prior](https://mlsanity.com/paper/2405.05749/) (2024)
- [DEGSTalk: Decomposed Per-Embedding Gaussian Fields for Hair-Preserving Talking Face Synthesis](https://mlsanity.com/paper/2412.20148/) (2024)
- [A Comprehensive Taxonomy and Analysis of Talking Head Synthesis: Techniques for Portrait Generation, Driving Mechanisms, and Editing](https://mlsanity.com/paper/2406.10553/) (2024)
- [SyncAnimation: A Real-Time End-to-End Framework for Audio-Driven Human Pose and Talking Head Animation](https://mlsanity.com/paper/2501.14646/) (2025)
- [Audio Driven Real-Time Facial Animation for Social Telepresence](https://mlsanity.com/paper/2510.01176/) (2025)
- [DT-NeRF: Decomposed Triplane-Hash Neural Radiance Fields for High-Fidelity Talking Portrait Synthesis](https://mlsanity.com/paper/2309.07752/) (2023)
- [PoseTalk: Text-and-Audio-based Pose Control and Motion Refinement for One-Shot Talking Head Generation](https://mlsanity.com/paper/2409.02657/) (2024)
- [GGTalker: Talking Head Systhesis with Generalizable Gaussian Priors and Identity-Specific Adaptation](https://mlsanity.com/paper/2506.21513/) (2025)
- [R2-Talker: Realistic Real-Time Talking Head Synthesis with Hash Grid Landmarks Encoding and Progressive Multilayer Conditioning](https://mlsanity.com/paper/2312.05572/) (2023)

## Abstract (arXiv)

This paper presents ER-NeRF, a novel conditional Neural Radiance Fields (NeRF) based architecture for talking portrait synthesis that can concurrently achieve fast convergence, real-time rendering, and state-of-the-art performance with small model size. Our idea is to explicitly exploit the unequal contribution of spatial regions to guide talking portrait modeling. Specifically, to improve the accuracy of dynamic head reconstruction, a compact and expressive NeRF-based Tri-Plane Hash Representation is introduced by pruning empty spatial regions with three planar hash encoders. For speech audio, we propose a Region Attention Module to generate region-aware condition feature via an attention mechanism. Different from existing methods that utilize an MLP-based encoder to learn the cross-modal relation implicitly, the attention mechanism builds an explicit connection between audio features and spatial regions to capture the priors of local motions. Moreover, a direct and fast Adaptive Pose Encoding is introduced to optimize the head-torso separation problem by mapping the complex transformation of the head pose into spatial coordinates. Extensive experiments demonstrate that our method renders better high-fidelity and audio-lips synchronized talking portrait videos, with realistic details and high efficiency compared to previous methods.