# EmoVOCA: Speech-Driven Emotional 3D Talking Heads

arXiv: https://arxiv.org/abs/2403.12886 · HTML: https://mlsanity.com/paper/2403.12886/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 32 references that are in the map.
- First posted on arXiv 19 Mar 2024.

## Builds on (most cited first, 19 of 19)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Capture, Learning, and Synthesis of 3D Speaking Styles](https://mlsanity.com/paper/1905.03079/) (2019)
- [EMOCA: Emotion Driven Monocular Face Capture and Animation](https://mlsanity.com/paper/2204.11312/) (2022)
- [FaceFormer: Speech-Driven 3D Facial Animation with Transformers](https://mlsanity.com/paper/2112.05329/) (2021)
- [MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement](https://mlsanity.com/paper/2104.08223/) (2021)
- [CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior](https://mlsanity.com/paper/2301.02379/) (2023)
- [EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animation](https://mlsanity.com/paper/2303.11089/) (2023)
- [Emotional Speech-Driven Animation with Content-Emotion Disentanglement](https://mlsanity.com/paper/2306.08990/) (2023)
- [Neural 3D Morphable Models: Spiral Convolutional Networks for 3D Shape Representation Learning and Generation](https://mlsanity.com/paper/1905.02876/) (2019)
- [FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusion](https://mlsanity.com/paper/2309.11306/) (2023)
- [Imitator: Personalized Speech-driven 3D Facial Animation](https://mlsanity.com/paper/2301.00023/) (2022)
- [SelfTalk: A Self-Supervised Commutative Training Diagram to Comprehend 3D Talking Faces](https://mlsanity.com/paper/2306.10799/) (2023)
- [SemanticStyleGAN: Learning Compositional Generative Priors for Controllable Image Synthesis and Editing](https://mlsanity.com/paper/2112.02236/) (2021)
- [LaughTalk: Expressive 3D Talking Head Generation with Laughter](https://mlsanity.com/paper/2311.00994/) (2023)
- [Sparse to Dense Dynamic 3D Facial Expression Generation](https://mlsanity.com/paper/2105.07463/) (2021)
- [Learning Landmarks Motion from Speech for Speaker-Agnostic 3D Talking Heads Generation](https://mlsanity.com/paper/2306.01415/) (2023)
- [The Florence 4D Facial Expression Dataset](https://mlsanity.com/paper/2210.16807/) (2022)
- [Audio-Driven 3D Facial Animation from In-the-Wild Videos](https://mlsanity.com/paper/2306.11541/) (2023)
- [Unsupervised Learning of Style-Aware Facial Animation from Real Acting Performances](https://mlsanity.com/paper/2306.10006/) (2023)

## Built on by (most cited first, 13 of 13)

- [A Review of Human Emotion Synthesis Based on Generative Technology](https://mlsanity.com/paper/2412.07116/) (2024)
- [MEDTalk: Multimodal Controlled 3D Facial Animation with Dynamic Emotions by Disentangled Embedding](https://mlsanity.com/paper/2507.06071/) (2025)
- [Beyond Fixed Topologies: Unregistered Training and Comprehensive Evaluation Metrics for 3D Talking Heads](https://mlsanity.com/paper/2410.11041/) (2024)
- [Supervising 3D Talking Head Avatars with Analysis-by-Audio-Synthesis](https://mlsanity.com/paper/2504.13386/) (2025)
- [JambaTalk: Speech-Driven 3D Talking Head Generation Based on Hybrid Transformer-Mamba Model](https://mlsanity.com/paper/2408.01627/) (2024)
- [LSF-Animation: Label-Free Speech-Driven Facial Animation via Implicit Feature Representation](https://mlsanity.com/paper/2510.21864/) (2025)
- [EmoTaG: Emotion-Aware Talking Head Synthesis on Gaussian Splatting with Few-Shot Personalization](https://mlsanity.com/paper/2603.21332/) (2026)
- [Think2Sing: Orchestrating Structured Motion Subtitles for Singing-Driven 3D Head Animation](https://mlsanity.com/paper/2509.02278/) (2025)
- [EmoDiffusion: Enhancing Emotional 3D Facial Animation with Latent Diffusion Models](https://mlsanity.com/paper/2503.11028/) (2025)
- [Learning Disentangled Speech- and Expression-Driven Blendshapes for 3D Talking Face Animation](https://mlsanity.com/paper/2510.25234/) (2025)
- [FreeTalk: Emotional Topology-Free 3D Talking Heads](https://mlsanity.com/paper/2603.15512/) (2026)
- [SEDTalker: Emotion-Aware 3D Facial Animation Using Frame-Level Speech Emotion Diarization](https://mlsanity.com/paper/2604.13335/) (2026)
- [Polyglot: Multilingual Style Preserving Speech-Driven Facial Animation](https://mlsanity.com/paper/2604.16108/) (2026)

## Abstract (arXiv)

The domain of 3D talking head generation has witnessed significant progress in recent years. A notable challenge in this field consists in blending speech-related motions with expression dynamics, which is primarily caused by the lack of comprehensive 3D datasets that combine diversity in spoken sentences with a variety of facial expressions. Whereas literature works attempted to exploit 2D video data and parametric 3D models as a workaround, these still show limitations when jointly modeling the two motions. In this work, we address this problem from a different perspective, and propose an innovative data-driven technique that we used for creating a synthetic dataset, called EmoVOCA, obtained by combining a collection of inexpressive 3D talking heads and a set of 3D expressive sequences. To demonstrate the advantages of this approach, and the quality of the dataset, we then designed and trained an emotional 3D talking head generator that accepts a 3D face, an audio file, an emotion label, and an intensity value as inputs, and learns to animate the audio-synchronized lip movements with expressive traits of the face. Comprehensive experiments, both quantitative and qualitative, using our data and generator evidence superior ability in synthesizing convincing animations, when compared with the best performing methods in the literature. Our code and pre-trained model will be made available.