mlsanityData to 14 Oct 2026

FaceFormer: Speech-Driven 3D Facial Animation with Transformers

arXiv 2112.05329 ↗First posted 10 Dec 2021cs.CV cs.GRcited by 209

Not on the map. Why?

Cited by 24 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Speech-driven 3D facial animation is challenging due to the complex geometry of human faces and the limited availability of 3D audio-visual data. Prior works typically focus on learning phoneme-level features of short audio windows with limited context, occasionally resulting in inaccurate lip movements. To tackle this limitation, we propose a Transformer-based autoregressive model, FaceFormer, which encodes the long-term audio context and autoregressively predicts a sequence of animated 3D face meshes. To cope with the data scarcity issue, we integrate the self-supervised pre-trained speech representations. Also, we devise two biased attention mechanisms well suited to this specific task, including the biased cross-modal multi-head (MH) attention and the biased causal MH self-attention with a periodic positional encoding strategy. The former effectively aligns the audio-motion modalities, whereas the latter offers abilities to generalize to longer audio sequences. Extensive experiments and a perceptual user study show that our approach outperforms the existing state-of-the-arts. The code will be made available.

Key facts. Cited by 209 papers in the map (data through 14 Oct 2026). Builds on 45 of its 72 references that are in the map. First posted on arXiv 10 Dec 2021.

Builds on

45 of 72 references.

References · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2020End-to-End Object Detection with Transformerscited by 7,414
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  7. 2020Longformer: The Long-Document Transformercited by 3,413
  8. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  9. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  10. 2020Point Transformercited by 1,470
  11. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  12. 2021Transformers in Vision: A Surveycited by 1,013
33 more
  1. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  2. 2018Image Transformercited by 772
  3. 2020PCT: Point cloud transformercited by 732
  4. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 672
  5. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 501
  6. 2020End-to-End Human Pose and Mesh Reconstruction with Transformerscited by 418
  7. 2021Conditional Positional Encodings for Vision Transformerscited by 406
  8. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  9. 2018Deep Video Portraitscited by 267
  10. 2019Capture, Learning, and Synthesis of 3D Speaking Stylescited by 259
  11. 2019Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Losscited by 252
  12. 2019Neural Voice Puppetry: Audio-driven Facial Reenactmentcited by 250
  13. 2018Talking Face Generation by Adversarially Disentangled Audio-Visual Representationcited by 212
  14. 2021MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglementcited by 190
  15. 2018X2Face: A network for controlling face generation by using images, audio, and pose codescited by 189
  16. 2021Audio-Driven Emotional Video Portraitscited by 180
  17. 2018Lip Movements Generation at a Glancecited by 155
  18. 2019Realistic Speech-Driven Facial Animation with GANscited by 142
  19. 2020A Spatio-temporal Transformer for 3D Human Motion Predictioncited by 133
  20. 2020Talking-head Generation with Rhythmic Head Motioncited by 112
  21. 2021Text2Gestures: A Transformer-Based Network for Generating Emotive Body Gestures for Virtual Agentscited by 106
  22. 2019Text-based Editing of Talking-head Videocited by 103
  23. 2017You said that?cited by 99
  24. 2021Learning Speech-driven 3D Conversational Gestures from Videocited by 98
  25. 2020Audio-driven Talking Face Video Generation with Learning-based Personalized Head Posecited by 92
  26. 2020Learning to Generate Diverse Dance Motions with Transformercited by 85
  27. 2021LipSync3D: Data-Efficient Learning of Personalized 3D Talking Faces from Video using Pose and Lighting Normalizationcited by 67
  28. 2018VisemeNet: Audio-Driven Animator-Centric Speech Animationcited by 63
  29. 2020Modality Dropout for Improved Performance-driven Talking Facescited by 32
  30. 2019Animating Face using Disentangled Audio Representationscited by 24
  31. 2021Transflower: probabilistic autoregressive dance generation with multimodal attentioncited by 22
  32. 2020Geometry-guided Dense Perspective Network for Speech-Driven Facial Animationcited by 19
  33. 20213D-TalkEmo: Learning to Synthesize 3D Emotional Talking Headcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 10; 2023: 51; 2024: 57; 2025: 44; 2026: 47.

209 papers.

Sort:

By area:

  1. 2022TEMOS: Generating diverse human motions from textual descriptionscited by 492
  2. 2022Multimodal Learning with Transformers: A Surveycited by 400
  3. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 382
  4. 2024EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditionscited by 235
  5. 2024LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Controlcited by 206
  6. 2024VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Timecited by 187
  7. 2023CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Priorcited by 184
  8. 2022Generating Holistic 3D Human Motion from Speechcited by 180
  9. 2023EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animationcited by 142
  10. 2023EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modelingcited by 105
  11. 2023Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expertcited by 98
  12. 2023Emotional Speech-Driven Animation with Content-Emotion Disentanglementcited by 98
38 more
  1. 2023StyleSync: High-Fidelity Generalized and Personalized Lip Sync in Style-based Generatorcited by 89
  2. 2023DiffPoseTalk: Speech-Driven Stylistic 3D Facial Animation and Head Pose Generation via Diffusion Modelscited by 81
  3. 2023FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusioncited by 78
  4. 2022Imitator: Personalized Speech-driven 3D Facial Animationcited by 74
  5. 2023SelfTalk: A Self-Supervised Commutative Training Diagram to Comprehend 3D Talking Facescited by 68
  6. 2024AniTalker: Animate Vivid and Diverse Talking Faces through Identity-Decoupled Facial Motion Encodingcited by 67
  7. 2024DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generationcited by 62
  8. 2023FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Modelscited by 55
  9. 2024VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesiscited by 50
  10. 2022SPACE: Speech-driven Portrait Animation with Controllable Expressioncited by 49
  11. 2023DAE-Talker: High Fidelity Speech-Driven Talking Face Generation with Diffusion Autoencodercited by 48
  12. 2024MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Modelscited by 46
  13. 2023LivelySpeaker: Towards Semantic-Aware Co-Speech Gesture Generationcited by 40
  14. 2024FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking Portraitcited by 40
  15. 2024Media2Face: Co-speech Facial Animation Generation With Multi-Modality Guidancecited by 39
  16. 2024Dyadic Interaction Modeling for Social Behavior Generationcited by 32
  17. 2024SemTalk: Holistic Co-speech Motion Generation with Frame-level Semantic Emphasiscited by 32
  18. 2025DualTalk: Dual-Speaker Interaction for 3D Talking Head Conversationscited by 32
  19. 2024UniTalker: Scaling up Audio-Driven 3D Facial Animation through A Unified Modelcited by 30
  20. 2023FaceXHuBERT: Text-less Speech-driven E(X)pressive 3D Facial Animation Synthesis Using Self-Supervised Speech Representation Learningcited by 29
  21. 2023MODA: Mapping-Once Audio-driven Portrait Animation with Dual Attentionscited by 29
  22. 2024Towards Variable and Coordinated Holistic Co-Speech Motion Generationcited by 29
  23. 2024MultiTalk: Enhancing 3D Talking Head Generation Across Languages with Multilingual Video Datasetcited by 28
  24. 2023LaughTalk: Expressive 3D Talking Head Generation with Laughtercited by 27
  25. 2025Human Motion Video Generation: A Surveycited by 25
  26. 2024ScanTalk: 3D Talking Heads from Unregistered Scanscited by 24
  27. 2022Deep Learning for Visual Speech Analysis: A Surveycited by 21
  28. 20243D Face Tracking from 2D Video through Iterative Dense UV to Image Flowcited by 20
  29. 2024DiffSpeaker: Speech-Driven 3D Facial Animation with Diffusion Transformercited by 18
  30. 2023ReactFace: Online Multiple Appropriate Facial Reaction Generation in Dyadic Interactionscited by 17
  31. 2023Speech-Driven 3D Face Animation with Composite and Regional Facial Movementscited by 17
  32. 2025EchoMask: Speech-Queried Attention-based Mask Modeling for Holistic Co-Speech Motion Generationcited by 17
  33. 2023DiffusionTalker: Personalization and Acceleration for Speech-Driven 3D Face Diffusercited by 16
  34. 2023Mimic: Speaking Style Disentanglement for Speech-Driven 3D Facial Animationcited by 16
  35. 2024Learn2Talk: 3D Talking Face Learns from 2D Talking Facecited by 16
  36. 2024ReSyncer: Rewiring Style-based Generator for Unified Audio-Visually Synced Facial Performercited by 16
  37. 2024MMHead: Towards Fine-grained Multi-modal 3D Facial Animationcited by 16
  38. 2023EMoG: Synthesizing Emotive Co-speech 3D Gesture with Diffusion Modelcited by 15

50 most cited of 209 All 209 (JSON)