# Joint Audio-Text Model for Expressive Speech-Driven 3D Facial Animation arXiv: https://arxiv.org/abs/2112.02214 ยท HTML: https://mlsanity.com/paper/2112.02214/ ## Key facts - Cited by 15 papers in the map (data through 14 Oct 2026). - Builds on 23 of its 45 references that are in the map. - First posted on arXiv 4 Dec 2021. ## Builds on (most cited first, 23 of 23) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021) - [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016) - [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015) - [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020) - [Tensor Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/1707.07250/) (2017) - [Deep Speech: Scaling up end-to-end speech recognition](https://mlsanity.com/paper/1412.5567/) (2014) - [Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation](https://mlsanity.com/paper/2104.11116/) (2021) - [Capture, Learning, and Synthesis of 3D Speaking Styles](https://mlsanity.com/paper/1905.03079/) (2019) - [Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Loss](https://mlsanity.com/paper/1905.03820/) (2019) - [Talking Face Generation by Adversarially Disentangled Audio-Visual Representation](https://mlsanity.com/paper/1807.07860/) (2018) - [Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identity](https://mlsanity.com/paper/2009.02119/) (2020) - [MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement](https://mlsanity.com/paper/2104.08223/) (2021) - [X2Face: A network for controlling face generation by using images, audio, and pose codes](https://mlsanity.com/paper/1807.10550/) (2018) - [Audio-Driven Emotional Video Portraits](https://mlsanity.com/paper/2104.07452/) (2021) - [Lip Movements Generation at a Glance](https://mlsanity.com/paper/1803.10404/) (2018) - [Realistic Speech-Driven Facial Animation with GANs](https://mlsanity.com/paper/1906.06337/) (2019) - [Talking-head Generation with Rhythmic Head Motion](https://mlsanity.com/paper/2007.08547/) (2020) - [Gesticulator: A framework for semantically-aware speech-driven gesture generation](https://mlsanity.com/paper/2001.09326/) (2020) - [VisemeNet: Audio-Driven Animator-Centric Speech Animation](https://mlsanity.com/paper/1805.09488/) (2018) - [Animating Face using Disentangled Audio Representations](https://mlsanity.com/paper/1910.00726/) (2019) - [3D-TalkEmo: Learning to Synthesize 3D Emotional Talking Head](https://mlsanity.com/paper/2104.12051/) (2021) ## Built on by (most cited first, 15 of 15) - [CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior](https://mlsanity.com/paper/2301.02379/) (2023) - [FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusion](https://mlsanity.com/paper/2309.11306/) (2023) - [Media2Face: Co-speech Facial Animation Generation With Multi-Modality Guidance](https://mlsanity.com/paper/2401.15687/) (2024) - [FaceXHuBERT: Text-less Speech-driven E(X)pressive 3D Facial Animation Synthesis Using Self-Supervised Speech Representation Learning](https://mlsanity.com/paper/2303.05416/) (2023) - [UniFLG: Unified Facial Landmark Generator from Text or Speech](https://mlsanity.com/paper/2302.14337/) (2023) - [Wav2Sem: Plug-and-Play Audio Semantic Decoupling for 3D Speech-Driven Facial Animation](https://mlsanity.com/paper/2505.23290/) (2025) - [InterAct: Capture and Modelling of Realistic, Expressive and Interactive Activities between Two Persons in Daily Scenarios](https://mlsanity.com/paper/2405.11690/) (2024) - [PMMTalk: Speech-Driven 3D Facial Animation from Complementary Pseudo Multi-modal Features](https://mlsanity.com/paper/2312.02781/) (2023) - [REACT2023: the first Multi-modal Multiple Appropriate Facial Reaction Generation Challenge](https://mlsanity.com/paper/2306.06583/) (2023) - [Towards High-fidelity 3D Talking Avatar with Personalized Dynamic Texture](https://mlsanity.com/paper/2503.00495/) (2025) - [DF-3DFace: One-to-Many Speech Synchronized 3D Face Animation with Diffusion](https://mlsanity.com/paper/2310.05934/) (2023) - [DualTalker: A Cross-Modal Dual Learning Approach for Speech-Driven 3D Facial Animation](https://mlsanity.com/paper/2311.04766/) (2023) - [Pose-Controllable 3D Facial Animation Synthesis using Hierarchical Audio-Vertex Attention](https://mlsanity.com/paper/2302.12532/) (2023) - [AudioFace: Language-Assisted Speech-Driven Facial Animation with Multimodal Language Models](https://mlsanity.com/paper/2605.07478/) (2026) - [ECHO: Dyadic 3D Facial Motion Generation with Asymmetric Deterministic Articulation and Stochastic Reaction](https://mlsanity.com/paper/2609.05506/) (2026) ## Abstract (arXiv) Speech-driven 3D facial animation with accurate lip synchronization has been widely studied. However, synthesizing realistic motions for the entire face during speech has rarely been explored. In this work, we present a joint audio-text model to capture the contextual information for expressive speech-driven 3D facial animation. The existing datasets are collected to cover as many different phonemes as possible instead of sentences, thus limiting the capability of the audio-based model to learn more diverse contexts. To address this, we propose to leverage the contextual text embeddings extracted from the powerful pre-trained language model that has learned rich contextual representations from large-scale text data. Our hypothesis is that the text features can disambiguate the variations in upper face expressions, which are not strongly correlated with the audio. In contrast to prior approaches which learn phoneme-level features from the text, we investigate the high-level contextual text features for speech-driven 3D facial animation. We show that the combined acoustic and textual modalities can synthesize realistic facial expressions while maintaining audio-lip synchronization. We conduct the quantitative and qualitative evaluations as well as the perceptual user study. The results demonstrate the superior performance of our model against existing state-of-the-art approaches.