mlsanityData to 14 Oct 2026

AudioFace: Language-Assisted Speech-Driven Facial Animation with Multimodal Language Models

arXiv 2605.07478 ↗First posted 8 May 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Speech-driven facial animation requires accurate correspondence between acoustic signals and facial motion, especially for articulation-related mouth movements. However, directly mapping speech audio to facial coefficients often overlooks the linguistic and phonetic structure underlying speech production. In this paper, we propose AudioFace, a language-assisted framework for speech-driven blendshape generation that treats mouth-related facial coefficient prediction as a structured generation problem guided by linguistic and articulatory information. Instead of relying solely on acoustic features, our method leverages the prior knowledge of multimodal large language models and introduces transcript- and phoneme-level cues to bridge speech signals with interpretable facial actions. Extensive experiments show that AudioFace achieves superior performance across multiple evaluation metrics, validating the effectiveness of language-assisted and multimodal-prior-guided speech-driven facial animation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 41 references that are in the map. First posted on arXiv 8 May 2026.

Builds on

30 of 41 references.

References · 30

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2025Qwen3-Omni Technical Reportcited by 497
  3. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 382
  4. 2021AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesiscited by 339
  5. 2019Capture, Learning, and Synthesis of 3D Speaking Stylescited by 259
  6. 2021FaceFormer: Speech-Driven 3D Facial Animation with Transformerscited by 209
  7. 2024EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditionscited by 196
  8. 2021MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglementcited by 190
  9. 2024VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Timecited by 187
  10. 2023CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Priorcited by 184
  11. 2021Audio-Driven Emotional Video Portraitscited by 180
  12. 2022BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesiscited by 178
18 more
  1. 2023EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animationcited by 142
  2. 2024Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependencycited by 102
  3. 2023Emotional Speech-Driven Animation with Content-Emotion Disentanglementcited by 98
  4. 2023FunASR: A Fundamental End-to-End Speech Recognition Toolkitcited by 95
  5. 2023Efficient Emotional Adaptation for Audio-Driven Talking-Head Generationcited by 81
  6. 2023FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusioncited by 78
  7. 2025LAM: Large Avatar Model for One-shot Animatable Gaussian Headcited by 43
  8. 2024Segment Any Text: A Universal Approach for Robust, Efficient and Adaptable Sentence Segmentationcited by 41
  9. 2024UniTalker: Scaling up Audio-Driven 3D Facial Animation through A Unified Modelcited by 30
  10. 2025Teller: Real-Time Streaming Audio-Driven Portrait Animation with Autoregressive Motion Generationcited by 16
  11. 2021Joint Audio-Text Model for Expressive Speech-Driven 3D Facial Animationcited by 15
  12. 2024KMTalk: Speech-Driven 3D Facial Animation with Key Motion Embeddingcited by 14
  13. 2023Learning Audio-Driven Viseme Dynamics for 3D Face Animationcited by 12
  14. 2022PaddleSpeech: An Easy-to-Use All-in-One Speech Toolkitcited by 9
  15. 2023SAiD: Speech-driven Blendshape Facial Animation with Diffusioncited by 8
  16. 2025Audio2Face-3D: Audio-driven Realistic Facial Animation For Digital Avatarscited by 7
  17. 2026SentiAvatar: Towards Expressive and Interactive Digital Humanscited by 4
  18. 2026SemanticFace: Semantic Facial Action Estimation via Semantic Distillation in Interpretable Spacecited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 20233DiFACE: Diffusion-based Speech-driven 3D Facial Animation and Editing
  2. 2025A Bridge from Audio to Video: Phoneme-Viseme Alignment Allows Every Face to Speak Multiple Languages
  3. 2024Diverse Code Query Learning for Speech-Driven Facial Animation
  4. 2024Media2Face: Co-speech Facial Animation Generation With Multi-Modality Guidance
  5. 20253DiFACE: Synthesizing and Editing Holistic 3D Facial Animation
  6. 2025KSDiff: Keyframe-Augmented Speech-Aware Dual-Path Diffusion for Facial Animation
  7. 2026ReFree: Towards Realistic Co-Speech Video Generation via Reward-Free RL and Multilevel Speech Guidance
  8. 2025Model See Model Do: Speech-Driven Facial Animation with Style Control