mlsanityData to 7 Oct 2026

MMFace4D: A Large-Scale Multi-Modal 4D Face Dataset for Audio-Driven 3D Face Animation

arXiv 2303.09797 ↗First posted 17 Mar 2023cs.CVcited by 8

Not on the map. Why?

Abstract (arXiv)

Audio-Driven Face Animation is an eagerly anticipated technique for applications such as VR/AR, games, and movie making. With the rapid development of 3D engines, there is an increasing demand for driving 3D faces with audio. However, currently available 3D face animation datasets are either scale-limited or quality-unsatisfied, which hampers further developments of audio-driven 3D face animation. To address this challenge, we propose MMFace4D, a large-scale multi-modal 4D (3D sequence) face dataset consisting of 431 identities, 35,904 sequences, and 3.9 million frames. MMFace4D exhibits two compelling characteristics: 1) a remarkably diverse set of subjects and corpus, encompassing actors spanning ages 15 to 68, and recorded sentences with durations ranging from 0.7 to 11.4 seconds. 2) It features synchronized audio and 3D mesh sequences with high-resolution face details. To capture the subtle nuances of 3D facial expressions, we leverage three synchronized RGBD cameras during the recording process. Upon MMFace4D, we construct a non-autoregressive framework for audio-driven 3D face animation. Our framework considers the regional and composite natures of facial animations, and surpasses contemporary state-of-the-art approaches both qualitatively and quantitatively. The code, model, and dataset will be publicly available.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 37 of its 70 references that are in the map. First posted on arXiv 17 Mar 2023.

Builds on

37 of 70 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  5. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 1,998
  6. 2019FaceForensics++: Learning to Detect Manipulated Facial Imagescited by 1,095
  7. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  8. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
  9. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 669
  10. 2019Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Setcited by 481
  11. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 449
  12. 2020Modular Primitives for High-Performance Differentiable Renderingcited by 439
25 more
  1. 2021AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesiscited by 339
  2. 2018Deep Audio-Visual Speech Recognitioncited by 318
  3. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 301
  4. 2019Capture, Learning, and Synthesis of 3D Speaking Stylescited by 257
  5. 2018Talking Face Generation by Adversarially Disentangled Audio-Visual Representationcited by 212
  6. 2021FaceFormer: Speech-Driven 3D Facial Animation with Transformerscited by 208
  7. 2020FaceScape: a Large-scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Predictioncited by 196
  8. 2021MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglementcited by 189
  9. 2023CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Priorcited by 183
  10. 2021Audio-Driven Emotional Video Portraitscited by 180
  11. 2022BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesiscited by 177
  12. 2018Lip Movements Generation at a Glancecited by 154
  13. 2022EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Modelcited by 151
  14. 2019Realistic Speech-Driven Facial Animation with GANscited by 142
  15. 2022DFA-NeRF: Personalized Talking Head Generation via Disentangled Face Attributes Neural Renderingcited by 80
  16. 2021LipSync3D: Data-Efficient Learning of Personalized 3D Talking Faces from Video using Pose and Lighting Normalizationcited by 67
  17. 2018Large-Scale Visual Speech Recognitioncited by 57
  18. 2021Imitating Arbitrary Talking Style for Realistic Audio-DrivenTalking Face Synthesiscited by 56
  19. 2018LRW-1000: A Naturally-Distributed Large-Scale Benchmark for Lip Reading in the Wildcited by 51
  20. 2020Speech Driven Talking Face Generation from a Single Image and an Emotion Conditioncited by 46
  21. 2022Emotion-Controllable Generalized Talking Face Generationcited by 43
  22. 2020Attention Mesh: High-fidelity Face Mesh Prediction in Real-timecited by 30
  23. 2022Audio-Driven Talking Face Video Generation with Dynamic Convolution Kernelscited by 23
  24. 2020Geometry-guided Dense Perspective Network for Speech-Driven Facial Animationcited by 19
  25. 2021AnyoneNet: Synchronized Speech and Talking Head Generation for Arbitrary Personcited by 17

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 1; 2025: 3.

8 papers.

  1. 2024UniTalker: Scaling up Audio-Driven 3D Facial Animation through A Unified Modelcited by 29
  2. 2023Speech-Driven 3D Face Animation with Composite and Regional Facial Movementscited by 17
  3. 2023SingingHead: A Large-scale 4D Dataset for Singing Head Animationcited by 9
  4. 2023PMMTalk: Speech-Driven 3D Facial Animation from Complementary Pseudo Multi-modal Featurescited by 6
  5. 2025Towards High-fidelity 3D Talking Avatar with Personalized Dynamic Texturecited by 5
  6. 2023Versatile Face Animator: Driving Arbitrary 3D Facial Avatar in RGBD Spacecited by 4
  7. 2025Express4D: Expressive, Friendly, and Extensible 4D Facial Motion Generation Benchmarkcited by 4
  8. 2025KeyframeFace: Language-Driven Facial Animation via Semantic Keyframescited by 0