# Audio2Face-3D: Audio-driven Realistic Facial Animation For Digital Avatars

arXiv: https://arxiv.org/abs/2508.16401 · HTML: https://mlsanity.com/paper/2508.16401/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 59 references that are in the map.
- First posted on arXiv 22 Aug 2025.

## Builds on (most cited first, 30 of 35)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis](https://mlsanity.com/paper/2103.11078/) (2021)
- [MakeItTalk: Speaker-Aware Talking-Head Animation](https://mlsanity.com/paper/2004.12992/) (2020)
- [Capture, Learning, and Synthesis of 3D Speaking Styles](https://mlsanity.com/paper/1905.03079/) (2019)
- [FaceFormer: Speech-Driven 3D Facial Animation with Transformers](https://mlsanity.com/paper/2112.05329/) (2021)
- [EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditions](https://mlsanity.com/paper/2407.08136/) (2024)
- [MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement](https://mlsanity.com/paper/2104.08223/) (2021)
- [Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motion](https://mlsanity.com/paper/2107.09293/) (2021)
- [DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animation](https://mlsanity.com/paper/2301.03786/) (2023)
- [GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesis](https://mlsanity.com/paper/2301.13430/) (2023)
- [StyleTalk: One-shot Talking Head Generation with Controllable Speaking Styles](https://mlsanity.com/paper/2301.01081/) (2023)
- [DiffPoseTalk: Speech-Driven Stylistic 3D Facial Animation and Head Pose Generation via Diffusion Models](https://mlsanity.com/paper/2310.00434/) (2023)
- [FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusion](https://mlsanity.com/paper/2309.11306/) (2023)
- [AniTalker: Animate Vivid and Diverse Talking Faces through Identity-Decoupled Facial Motion Encoding](https://mlsanity.com/paper/2405.03121/) (2024)
- [FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models](https://mlsanity.com/paper/2312.08459/) (2023)
- [TalkCLIP: Talking Head Generation with Text-Guided Expressive Speaking Styles](https://mlsanity.com/paper/2304.00334/) (2023)
- [FaceXHuBERT: Text-less Speech-driven E(X)pressive 3D Facial Animation Synthesis Using Self-Supervised Speech Representation Learning](https://mlsanity.com/paper/2303.05416/) (2023)
- [Phone-to-audio alignment without text: A Semi-supervised Approach](https://mlsanity.com/paper/2110.03876/) (2021)

## Built on by (most cited first, 7 of 7)

- [Ex-Omni: Enabling 3D Facial Animation Generation for Omni-modal Large Language Models](https://mlsanity.com/paper/2602.07106/) (2026)
- [KeyframeFace: Language-Driven Facial Animation via Semantic Keyframes](https://mlsanity.com/paper/2512.11321/) (2025)
- [SingingBot: An Avatar-Driven System for Robotic Face Singing Performance](https://mlsanity.com/paper/2601.02125/) (2026)
- [AudioFace: Language-Assisted Speech-Driven Facial Animation with Multimodal Language Models](https://mlsanity.com/paper/2605.07478/) (2026)
- [Voice "Cloning" is Style Transfer](https://mlsanity.com/paper/2605.16578/) (2026)
- [Toward Accessible Psychotherapy Training Using AI-Driven Interactive Patient Avatars](https://mlsanity.com/paper/2606.17786/) (2026)
- [Mind Companion: An Embodied Conversational Agent for Process-Based Psychotherapy](https://mlsanity.com/paper/2606.17789/) (2026)

## Abstract (arXiv)

Audio-driven facial animation presents an effective solution for animating digital avatars. In this paper, we detail the technical aspects of NVIDIA Audio2Face-3D, including data acquisition, network architecture, retargeting methodology, evaluation metrics, and use cases. Audio2Face-3D system enables real-time interaction between human users and interactive avatars, facilitating facial animation authoring for game characters. To assist digital avatar creators and game developers in generating realistic facial animations, we have open-sourced Audio2Face-3D networks, SDK, training framework, and example dataset.