# GaussianSpeech: Audio-Driven Gaussian Avatars

arXiv: https://arxiv.org/abs/2411.18675 · HTML: https://mlsanity.com/paper/2411.18675/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 74 references that are in the map.
- First posted on arXiv 27 Nov 2024.

## Builds on (most cited first, 8 of 8)

- [GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians](https://mlsanity.com/paper/2312.02069/) (2023)
- [Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation](https://mlsanity.com/paper/2301.03396/) (2023)
- [GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesis](https://mlsanity.com/paper/2301.13430/) (2023)
- [Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation](https://mlsanity.com/paper/2201.07786/) (2022)
- [Talking-head Generation with Rhythmic Head Motion](https://mlsanity.com/paper/2007.08547/) (2020)
- [Real-time Neural Radiance Talking Portrait Synthesis via Audio-spatial Decomposition](https://mlsanity.com/paper/2211.12368/) (2022)
- [DFA-NeRF: Personalized Talking Head Generation via Disentangled Face Attributes Neural Rendering](https://mlsanity.com/paper/2201.00791/) (2022)
- [Memories are One-to-Many Mapping Alleviators in Talking Face Generation](https://mlsanity.com/paper/2212.05005/) (2022)

## Built on by (most cited first, 9 of 9)

- [AV-Flow: Transforming Text to Audio-Visual Human-like Interactions](https://mlsanity.com/paper/2502.13133/) (2025)
- [GGTalker: Talking Head Systhesis with Generalizable Gaussian Priors and Identity-Specific Adaptation](https://mlsanity.com/paper/2506.21513/) (2025)
- [GaussianHeadTalk: Wobble-Free 3D Talking Heads with Audio Driven Gaussian Splatting](https://mlsanity.com/paper/2512.10939/) (2025)
- [Few-Shot Identity Adaptation for 3D Talking Heads via Global Gaussian Field](https://mlsanity.com/paper/2506.22044/) (2025)
- [PASE: Phoneme-Aware Speech Encoder to Improve Lip Sync Accuracy for Talking Head Synthesis](https://mlsanity.com/paper/2504.05803/) (2025)
- [Splat-Portrait: Generalizing Talking Heads with Gaussian Splatting](https://mlsanity.com/paper/2601.18633/) (2026)
- [Giving Faces Their Feelings Back: Explicit Emotion Control for Feedforward Single-Image 3D Head Avatars](https://mlsanity.com/paper/2604.14541/) (2026)
- [FFAvatar: Feed-Forward 4D Head Avatar Reconstruction from Sparse Portrait Images](https://mlsanity.com/paper/2606.30347/) (2026)
- [GaussianEmoTalker: Real-Time Emotional Talking Head Synthesis with Audio-Driven and Blendshape-Based 3D Gaussian Splatting](https://mlsanity.com/paper/2607.00959/) (2026)

## Abstract (arXiv)

We introduce GaussianSpeech, a novel approach that synthesizes high-fidelity animation sequences of photo-realistic, personalized 3D human head avatars from spoken audio. To capture the expressive, detailed nature of human heads, including skin furrowing and finer-scale facial movements, we propose to couple speech signal with 3D Gaussian splatting to create realistic, temporally coherent motion sequences. We propose a compact and efficient 3DGS-based avatar representation that generates expression-dependent color and leverages wrinkle- and perceptually-based losses to synthesize facial details, including wrinkles that occur with different expressions. To enable sequence modeling of 3D Gaussian splats with audio, we devise an audio-conditioned transformer model capable of extracting lip and expression features directly from audio input. Due to the absence of high-quality datasets of talking humans in correspondence with audio, we captured a new large-scale multi-view dataset of audio-visual sequences of talking humans with native English accents and diverse facial geometry. GaussianSpeech consistently achieves state-of-the-art performance with visually natural motion at real time rendering rates, while encompassing diverse facial expressions and styles.