mlsanityData to 14 Oct 2026

MEDTalk: Multimodal Controlled 3D Facial Animation with Dynamic Emotions by Disentangled Embedding

arXiv 2507.06071 ↗First posted 8 Jul 2025cs.CV cs.MMcited by 6

Not on the map. Why?

Abstract (arXiv)

Audio-driven emotional 3D facial animation aims to generate synchronized lip movements and vivid facial expressions. However, most existing approaches focus on static and predefined emotion labels, limiting their diversity and naturalness. To address these challenges, we propose MEDTalk, a novel framework for fine-grained and dynamic emotional talking head generation. Our approach first disentangles content and emotion embedding spaces from motion sequences using a carefully designed cross-reconstruction process, enabling independent control over lip movements and facial expressions. Beyond conventional audio-driven lip synchronization, we integrate audio and speech text, predicting frame-wise intensity variations and dynamically adjusting static emotion features to generate realistic emotional expressions. Furthermore, to enhance control and personalization, we incorporate multimodal inputs-including text descriptions and reference expression images-to guide the generation of user-specified facial expressions. With MetaHuman as the priority, our generated results can be conveniently integrated into the industrial production pipeline. The code is available at: https://github.com/SJTU-Lucy/MEDTalk.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 36 of its 56 references that are in the map. First posted on arXiv 8 Jul 2025.

Builds on

36 of 56 references.

References · 36

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  4. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,917
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  6. 2019MediaPipe: A Framework for Building Perception Pipelinescited by 583
  7. 2020Revisiting Pre-Trained Models for Chinese Natural Language Processingcited by 266
  8. 2020CLUB: A Contrastive Log-ratio Upper Bound of Mutual Informationcited by 235
  9. 2021FaceFormer: Speech-Driven 3D Facial Animation with Transformerscited by 209
  10. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 197
  11. 2021MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglementcited by 190
  12. 2023CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Priorcited by 184
24 more
  1. 2021Audio-Driven Emotional Video Portraitscited by 180
  2. 2023EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animationcited by 142
  3. 2023Emotional Speech-Driven Animation with Content-Emotion Disentanglementcited by 98
  4. 2023Efficient Emotional Adaptation for Audio-Driven Talking-Head Generationcited by 81
  5. 2023DiffPoseTalk: Speech-Driven Stylistic 3D Facial Animation and Head Pose Generation via Diffusion Modelscited by 81
  6. 2023FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusioncited by 78
  7. 2023SelfTalk: A Self-Supervised Commutative Training Diagram to Comprehend 3D Talking Facescited by 68
  8. 2018VisemeNet: Audio-Driven Animator-Centric Speech Animationcited by 63
  9. 2024Media2Face: Co-speech Facial Animation Generation With Multi-Modality Guidancecited by 39
  10. 2024FlowVQTalker: High-Quality Emotional Talking Face Generation through Normalizing Flow and Quantizationcited by 39
  11. 2023High-fidelity Generalized Emotional Talking Face Generation with Multi-modal Emotion Space Learningcited by 37
  12. 2024Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Stylecited by 33
  13. 2024Say Anything with Any Stylecited by 26
  14. 2023DiffusionTalker: Personalization and Acceleration for Speech-Driven 3D Face Diffusercited by 16
  15. 2024MMHead: Towards Fine-grained Multi-modal 3D Facial Animationcited by 16
  16. 2025FixTalk: Taming Identity Leakage for High-Quality Talking Head Generation in Extreme Casescited by 14
  17. 2023ExpCLIP: Bridging Text and Facial Expressions via Semantic Alignmentcited by 13
  18. 2024EmoVOCA: Speech-Driven Emotional 3D Talking Headscited by 13
  19. 2020Learning Disentangled Phone and Speaker Representations in a Semi-Supervised VQ-VAE Paradigmcited by 12
  20. 2024AVI-Talking: Learning Audio-Visual Instructions for Expressive 3D Talking Face Generationcited by 9
  21. 2024EmoFace: Audio-driven Emotional 3D Face Animationcited by 9
  22. 2023PMMTalk: Speech-Driven 3D Facial Animation from Complementary Pseudo Multi-modal Featurescited by 6
  23. 2022A Survey on Applications of Digital Human Avatars toward Virtual Co-presencecited by 4
  24. 2023Breathing Life into Faces: Speech-driven 3D Facial Animation with Natural Head Pose and Detailed Shapecited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026FreeTalk: Emotional Topology-Free 3D Talking Headscited by 0
  2. 2026Personalizing Causal Audio-Driven Facial Motion via Dynamic Multi-modal Retrievalcited by 0
  3. 2026CapTalk: Text-Guided Stylization and Speech-Driven 3D Head Animationcited by 0
  4. 2026TokTalk: Expressive Real-time Facial Animation from Audio-LLM Tokenscited by 0
  5. 2026KM-Speaker: Keypoint-Based Style Control for High-Quality Speech-Driven 3D Facial Animation and Dialogue Localizationcited by 0
  6. 2026SubtleTalk: Generating Controllable Weakly-correlated Facial Dynamics for 3D Talking Heads via Residual Flow Matchingcited by 0