mlsanityData to 14 Oct 2026

Wav2Sem: Plug-and-Play Audio Semantic Decoupling for 3D Speech-Driven Facial Animation

arXiv 2505.23290 ↗First posted 29 May 2025cs.SD cs.CV eess.AScited by 8

Not on the map. Why?

Abstract (arXiv)

In 3D speech-driven facial animation generation, existing methods commonly employ pre-trained self-supervised audio models as encoders. However, due to the prevalence of phonetically similar syllables with distinct lip shapes in language, these near-homophone syllables tend to exhibit significant coupling in self-supervised audio feature spaces, leading to the averaging effect in subsequent lip motion generation. To address this issue, this paper proposes a plug-and-play semantic decorrelation module-Wav2Sem. This module extracts semantic features corresponding to the entire audio sequence, leveraging the added semantic information to decorrelate audio encodings within the feature space, thereby achieving more expressive audio features. Extensive experiments across multiple Speech-driven models indicate that the Wav2Sem module effectively decouples audio features, significantly alleviating the averaging effect of phonetically similar syllables in lip shape generation, thereby enhancing the precision and naturalness of facial animations. Our source code is available at https://github.com/wslh852/Wav2Sem.git.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 33 of its 46 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

33 of 46 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  5. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 780
  6. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 672
  7. 2014Deep Speech: Scaling up end-to-end speech recognitioncited by 354
  8. 2019Capture, Learning, and Synthesis of 3D Speaking Stylescited by 259
  9. 2019Neural Voice Puppetry: Audio-driven Facial Reenactmentcited by 250
  10. 2021FaceFormer: Speech-Driven 3D Facial Animation with Transformerscited by 209
  11. 2021MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglementcited by 190
  12. 2023CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Priorcited by 184
21 more
  1. 2023GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latentscited by 158
  2. 2023EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animationcited by 142
  3. 2023EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modelingcited by 105
  4. 2023Emotional Speech-Driven Animation with Content-Emotion Disentanglementcited by 98
  5. 2021One-shot Talking Face Generation from Single-speaker Audio-Visual Correlation Learningcited by 87
  6. 2023DiffPoseTalk: Speech-Driven Stylistic 3D Facial Animation and Head Pose Generation via Diffusion Modelscited by 81
  7. 2023FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusioncited by 78
  8. 2022Imitator: Personalized Speech-driven 3D Facial Animationcited by 74
  9. 2023FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Modelscited by 55
  10. 2023OMG: Towards Open-vocabulary Motion Generation via Mixture of Controllerscited by 47
  11. 2024ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesiscited by 41
  12. 2023Emotional Speech-driven 3D Body Animation via Disentangled Latent Diffusioncited by 39
  13. 2024Media2Face: Co-speech Facial Animation Generation With Multi-Modality Guidancecited by 39
  14. 2024UniTalker: Scaling up Audio-Driven 3D Facial Animation through A Unified Modelcited by 30
  15. 2023FaceXHuBERT: Text-less Speech-driven E(X)pressive 3D Facial Animation Synthesis Using Self-Supervised Speech Representation Learningcited by 29
  16. 2022Mask-FPAN: Semi-Supervised Face Parsing in the Wild With De-Occlusion and UV GANcited by 16
  17. 2023Mimic: Speaking Style Disentanglement for Speech-Driven 3D Facial Animationcited by 16
  18. 2021Joint Audio-Text Model for Expressive Speech-Driven 3D Facial Animationcited by 15
  19. 20233DiFACE: Diffusion-based Speech-driven 3D Facial Animation and Editingcited by 15
  20. 2021Talking Head Generation with Audio and Speech Related Facial Action Unitscited by 12
  21. 2024You Only Speak Once to Seecited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 6.

8 papers.

  1. 2025Mitigating Error Accumulation in Co-Speech Motion Generation via Global Rotation Diffusion and Multi-Level Constraintscited by 12
  2. 2026RAM: Recover Any 3D Human Motion in-the-Wildcited by 12
  3. 2026Integrating Facial Generation into Full-Duplex Spoken Dialogue Systemscited by 2
  4. 2025CNSocialDepress: A Chinese Social Media Dataset for Depression Risk Detection and Structured Analysiscited by 1
  5. 2026A2-LLM: An End-to-end Conversational Audio Avatar Large Language Modelcited by 1
  6. 2026AnyTalk: Speech Animation for Arbitrary Characters Leveraging a Video Generation Modelcited by 0
  7. 2026ECHO: Dyadic 3D Facial Motion Generation with Asymmetric Deterministic Articulation and Stochastic Reactioncited by 0
  8. 2026Seeing Speech: Learning Visible Articulatory Dynamics for Speech-Driven 3D Facial Animationcited by 0