# CSTalk: Correlation Supervised Speech-driven 3D Emotional Facial Animation Generation

arXiv: https://arxiv.org/abs/2404.18604 · HTML: https://mlsanity.com/paper/2404.18604/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 22 references that are in the map.
- First posted on arXiv 29 Apr 2024.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [Capture, Learning, and Synthesis of 3D Speaking Styles](https://mlsanity.com/paper/1905.03079/) (2019)
- [FaceFormer: Speech-Driven 3D Facial Animation with Transformers](https://mlsanity.com/paper/2112.05329/) (2021)
- [MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement](https://mlsanity.com/paper/2104.08223/) (2021)
- [EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animation](https://mlsanity.com/paper/2303.11089/) (2023)
- [Learning Audio-Driven Viseme Dynamics for 3D Face Animation](https://mlsanity.com/paper/2301.06059/) (2023)
- [3D-TalkEmo: Learning to Synthesize 3D Emotional Talking Head](https://mlsanity.com/paper/2104.12051/) (2021)

## Built on by (most cited first, 2 of 2)

- [Few-Shot Identity Adaptation for 3D Talking Heads via Global Gaussian Field](https://mlsanity.com/paper/2506.22044/) (2025)
- [EditEmoTalk: Controllable Speech-Driven 3D Facial Animation with Continuous Expression Editing](https://mlsanity.com/paper/2601.10000/) (2026)

## Abstract (arXiv)

Speech-driven 3D facial animation technology has been developed for years, but its practical application still lacks expectations. The main challenges lie in data limitations, lip alignment, and the naturalness of facial expressions. Although lip alignment has seen many related studies, existing methods struggle to synthesize natural and realistic expressions, resulting in a mechanical and stiff appearance of facial animations. Even with some research extracting emotional features from speech, the randomness of facial movements limits the effective expression of emotions. To address this issue, this paper proposes a method called CSTalk (Correlation Supervised) that models the correlations among different regions of facial movements and supervises the training of the generative model to generate realistic expressions that conform to human facial motion patterns. To generate more intricate animations, we employ a rich set of control parameters based on the metahuman character model and capture a dataset for five different emotions. We train a generative network using an autoencoder structure and input an emotion embedding vector to achieve the generation of user-control expressions. Experimental results demonstrate that our method outperforms existing state-of-the-art methods.