mlsanityData to 7 Oct 2026

VisemeNet: Audio-Driven Animator-Centric Speech Animation

arXiv 1805.09488 ↗First posted 24 May 2018cs.GRcited by 62

Not on the map. Why?

Abstract (arXiv)

We present a novel deep-learning based approach to producing animator-centric speech motion curves that drive a JALI or standard FACS-based production face-rig, directly from input audio. Our three-stage Long Short-Term Memory (LSTM) network architecture is motivated by psycho-linguistic insights: segmenting speech audio into a stream of phonetic-groups is sufficient for viseme construction; speech styles like mumbling or shouting are strongly co-related to the motion of facial landmarks; and animator style is encoded in viseme motion curve profiles. Our contribution is an automatic real-time lip-synchronization from audio solution that integrates seamlessly into existing animation pipelines. We evaluate our results by: cross-validation to ground-truth data; animator critique and edits; visual comparison to recent deep-learning lip-synchronization solutions; and showing our approach to be resilient to diversity in speaker and language.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 May 2018.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 13; 2022: 12; 2023: 23; 2024: 4; 2025: 10.

62 papers.

Sort:

By area:

  1. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  2. 2021FaceFormer: Speech-Driven 3D Facial Animation with Transformerscited by 208
  3. 2021MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglementcited by 189
  4. 2023CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Priorcited by 183
  5. 2021Audio-Driven Emotional Video Portraitscited by 180
  6. 2022Generating Holistic 3D Human Motion from Speechcited by 180
  7. 2022EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Modelcited by 151
  8. 2025MUSt3R: Multi-view Network for Stereo 3D Reconstructioncited by 139
  9. 2023Emotional Speech-Driven Animation with Content-Emotion Disentanglementcited by 98
  10. 2023StyleSync: High-Fidelity Generalized and Personalized Lip Sync in Style-based Generatorcited by 89
  11. 2022DFA-NeRF: Personalized Talking Head Generation via Disentangled Face Attributes Neural Renderingcited by 80
  12. 2023FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusioncited by 77
38 more
  1. 2021LipSync3D: Data-Efficient Learning of Personalized 3D Talking Faces from Video using Pose and Lighting Normalizationcited by 67
  2. 2023SelfTalk: A Self-Supervised Commutative Training Diagram to Comprehend 3D Talking Facescited by 67
  3. 2022SPACE: Speech-driven Portrait Animation with Controllable Expressioncited by 49
  4. 2024Media2Face: Co-speech Facial Animation Generation With Multi-Modality Guidancecited by 39
  5. 2021Live Speech Portraits: Real-Time Photorealistic Talking-Head Animationcited by 36
  6. 2025DualTalk: Dual-Speaker Interaction for 3D Talking Head Conversationscited by 31
  7. 2023FaceXHuBERT: Text-less Speech-driven E(X)pressive 3D Facial Animation Synthesis Using Self-Supervised Speech Representation Learningcited by 29
  8. 2022Audio-driven Neural Gesture Reenactment with Video Motion Graphscited by 18
  9. 2023Speech-Driven 3D Face Animation with Composite and Regional Facial Movementscited by 17
  10. 2023DiffusionTalker: Personalization and Acceleration for Speech-Driven 3D Face Diffusercited by 16
  11. 2021Joint Audio-Text Model for Expressive Speech-Driven 3D Facial Animationcited by 15
  12. 2023Probabilistic Speech-Driven 3D Facial Motion Synthesis: New Benchmarks, Methods, and Applicationscited by 15
  13. 2021Multi-modality Deep Restoration of Extremely Compressed Face Videoscited by 14
  14. 2023Learning Audio-Driven Viseme Dynamics for 3D Face Animationcited by 12
  15. 20213D-TalkEmo: Learning to Synthesize 3D Emotional Talking Headcited by 10
  16. 2023SingingHead: A Large-scale 4D Dataset for Singing Head Animationcited by 9
  17. 2024EmoFace: Audio-driven Emotional 3D Face Animationcited by 9
  18. 2023SAiD: Speech-driven Blendshape Facial Animation with Diffusioncited by 8
  19. 2025OT-Talk: Animating 3D Talking Head with Optimal Transportationcited by 6
  20. 2025Advancing Talking Head Generation: A Comprehensive Survey of Multi-Modal Methodologies, Datasets, Evaluation Metrics, and Loss Functionscited by 6
  21. 2025MEDTalk: Multimodal Controlled 3D Facial Animation with Dynamic Emotions by Disentangled Embeddingcited by 6
  22. 2023NeuBTF: Neural fields for BTF encoding and transfercited by 5
  23. 2025Supervising 3D Talking Head Avatars with Analysis-by-Audio-Synthesiscited by 5
  24. 2022Synthesizing Photorealistic Virtual Humans Through Cross-modal Disentanglementcited by 4
  25. 2023Expressive Speech-driven Facial Animation with controllable emotionscited by 4
  26. 2023Single-View View Synthesis with Self-Rectified Pseudo-Stereocited by 4
  27. 2023AVFace: Towards Detailed Audio-Visual 4D Face Reconstructioncited by 4
  28. 2023Style Transfer for 2D Talking Head Animationcited by 3
  29. 2023DF-3DFace: One-to-Many Speech Synchronized 3D Face Animation with Diffusioncited by 3
  30. 2023Breathing Life into Faces: Speech-driven 3D Facial Animation with Natural Head Pose and Detailed Shapecited by 3
  31. 2025LSF-Animation: Label-Free Speech-Driven Facial Animation via Implicit Feature Representationcited by 3
  32. 2023Unsupervised Learning of Style-Aware Facial Animation from Real Acting Performancescited by 2
  33. 2023RADIO: Reference-Agnostic Dubbing Video Synthesiscited by 2
  34. 2023DualTalker: A Cross-Modal Dual Learning Approach for Speech-Driven 3D Facial Animationcited by 2
  35. 2024ID-NeRF: Indirect Diffusion-guided Neural Radiance Fields for Generalizable View Synthesiscited by 2
  36. 2025Single-image Reflectance and Transmittance Estimation from Any Flatbed Scannercited by 1
  37. 2025PTalker: Personalized Speech-Driven 3D Talking Head Animation via Style Disentanglement and Modality Alignmentcited by 1
  38. 2021AudioVisual Speech Synthesis: A brief literature reviewcited by 0

50 most cited of 62 All 62 (JSON)