# ExpCLIP: Bridging Text and Facial Expressions via Semantic Alignment

arXiv: https://arxiv.org/abs/2308.14448 · HTML: https://mlsanity.com/paper/2308.14448/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 42 references that are in the map.
- First posted on arXiv 28 Aug 2023.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [MotionCLIP: Exposing Human Motion Generation to CLIP Space](https://mlsanity.com/paper/2203.08063/) (2022)
- [Capture, Learning, and Synthesis of 3D Speaking Styles](https://mlsanity.com/paper/1905.03079/) (2019)
- [FaceFormer: Speech-Driven 3D Facial Animation with Transformers](https://mlsanity.com/paper/2112.05329/) (2021)
- [MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement](https://mlsanity.com/paper/2104.08223/) (2021)
- [CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior](https://mlsanity.com/paper/2301.02379/) (2023)
- [Audio-Driven Emotional Video Portraits](https://mlsanity.com/paper/2104.07452/) (2021)
- [BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesis](https://mlsanity.com/paper/2203.05297/) (2022)
- [ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning Challenges](https://mlsanity.com/paper/2202.10659/) (2022)
- [EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Model](https://mlsanity.com/paper/2205.15278/) (2022)
- [Emotional Speech-Driven Animation with Content-Emotion Disentanglement](https://mlsanity.com/paper/2306.08990/) (2023)
- [Progressive Disentangled Representation Learning for Fine-Grained Controllable Talking Head Synthesis](https://mlsanity.com/paper/2211.14506/) (2022)
- [StyleTalk: One-shot Talking Head Generation with Controllable Speaking Styles](https://mlsanity.com/paper/2301.01081/) (2023)
- [Imitating Arbitrary Talking Style for Realistic Audio-DrivenTalking Face Synthesis](https://mlsanity.com/paper/2111.00203/) (2021)
- [Emotion-Controllable Generalized Talking Face Generation](https://mlsanity.com/paper/2205.01155/) (2022)
- [A Hierarchical Representation Network for Accurate and Detailed Face Reconstruction from In-The-Wild Images](https://mlsanity.com/paper/2302.14434/) (2023)
- [Emotion Intensities in Tweets](https://mlsanity.com/paper/1708.03696/) (2017)
- [HiFace: High-Fidelity 3D Face Reconstruction by Learning Static and Dynamic Details](https://mlsanity.com/paper/2303.11225/) (2023)
- [Speech-Driven Expressive Talking Lips with Conditional Sequential Generative Adversarial Networks](https://mlsanity.com/paper/1806.00154/) (2018)
- [StyleCLIPDraw: Coupling Content and Style in Text-to-Drawing Synthesis](https://mlsanity.com/paper/2111.03133/) (2021)
- [Transformer-S2A: Robust and Efficient Speech-to-Animation](https://mlsanity.com/paper/2111.09771/) (2021)

## Built on by (most cited first, 13 of 13)

- [Emotional Speech-driven 3D Body Animation via Disentangled Latent Diffusion](https://mlsanity.com/paper/2312.04466/) (2023)
- [InstructAvatar: Text-Guided Emotion and Motion Control for Avatar Generation](https://mlsanity.com/paper/2405.15758/) (2024)
- [MMHead: Towards Fine-grained Multi-modal 3D Facial Animation](https://mlsanity.com/paper/2410.07757/) (2024)
- [SingingHead: A Large-scale 4D Dataset for Singing Head Animation](https://mlsanity.com/paper/2312.04369/) (2023)
- [Audio2Face-3D: Audio-driven Realistic Facial Animation For Digital Avatars](https://mlsanity.com/paper/2508.16401/) (2025)
- [MEDTalk: Multimodal Controlled 3D Facial Animation with Dynamic Emotions by Disentangled Embedding](https://mlsanity.com/paper/2507.06071/) (2025)
- [Think2Sing: Orchestrating Structured Motion Subtitles for Singing-Driven 3D Head Animation](https://mlsanity.com/paper/2509.02278/) (2025)
- [EmoCAST: Emotional Talking Portrait via Emotive Text Description](https://mlsanity.com/paper/2508.20615/) (2025)
- [EmoteGPT: 3D Human Facial Expressions from Natural Language Descriptions](https://mlsanity.com/paper/2607.02674/) (2026)
- [Emo3D: Metric and Benchmarking Dataset for 3D Facial Expression Generation from Emotion Description](https://mlsanity.com/paper/2410.02049/) (2024)
- [KeyframeFace: Language-Driven Facial Animation via Semantic Keyframes](https://mlsanity.com/paper/2512.11321/) (2025)
- [KM-Speaker: Keypoint-Based Style Control for High-Quality Speech-Driven 3D Facial Animation and Dialogue Localization](https://mlsanity.com/paper/2606.28568/) (2026)
- [SubtleTalk: Generating Controllable Weakly-correlated Facial Dynamics for 3D Talking Heads via Residual Flow Matching](https://mlsanity.com/paper/2608.06408/) (2026)

## Abstract (arXiv)

The objective of stylized speech-driven facial animation is to create animations that encapsulate specific emotional expressions. Existing methods often depend on pre-established emotional labels or facial expression templates, which may limit the necessary flexibility for accurately conveying user intent. In this research, we introduce a technique that enables the control of arbitrary styles by leveraging natural language as emotion prompts. This technique presents benefits in terms of both flexibility and user-friendliness. To realize this objective, we initially construct a Text-Expression Alignment Dataset (TEAD), wherein each facial expression is paired with several prompt-like descriptions.We propose an innovative automatic annotation method, supported by Large Language Models (LLMs), to expedite the dataset construction, thereby eliminating the substantial expense of manual annotation. Following this, we utilize TEAD to train a CLIP-based model, termed ExpCLIP, which encodes text and facial expressions into semantically aligned style embeddings. The embeddings are subsequently integrated into the facial animation generator to yield expressive and controllable facial animations. Given the limited diversity of facial emotions in existing speech-driven facial animation training data, we further introduce an effective Expression Prompt Augmentation (EPA) mechanism to enable the animation generator to support unprecedented richness in style control. Comprehensive experiments illustrate that our method accomplishes expressive facial animation generation and offers enhanced flexibility in effectively conveying the desired style.