mlsanityData to 7 Oct 2026

Data standardization for robust lip sync

arXiv 2202.06198 ↗First posted 13 Feb 2022cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Lip sync is a fundamental audio-visual task. However, existing lip sync methods fall short of being robust in the wild. One important cause could be distracting factors on the visual input side, making extracting lip motion information difficult. To address these issues, this paper proposes a data standardization pipeline to standardize the visual input for lip sync. Based on recent advances in 3D face reconstruction, we first create a model that can consistently disentangle lip motion information from the raw images. Then, standardized images are synthesized with disentangled lip motion information, with all other attributes related to distracting factors set to predefined values independent of the input, to reduce their effects. Using synthesized images, existing lip sync methods improve their data efficiency and robustness, and they achieve competitive performance for the active speaker detection task.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 17 of its 30 references that are in the map. First posted on arXiv 13 Feb 2022.

Builds on

17 of 30 references.

References · 17

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  4. 2020Face2Face: Real-time Face Capture and Reenactment of RGB Videoscited by 631
  5. 2019Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Setcited by 481
  6. 20193D Morphable Face Models -- Past, Present and Futurecited by 269
  7. 2019AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detectioncited by 81
  8. 2018FML: Face Model Learning from Videoscited by 78
  9. 2020Self-Supervised Monocular 3D Face Reconstruction by Occlusion-Aware Multi-view Geometry Consistencycited by 78
  10. 2018Perfect match: Improved cross-modal embeddings for audio-visual synchronisationcited by 60
  11. 2020Rotate-and-Render: Unsupervised Photorealistic Face Rotation from Single-View Imagescited by 37
  12. 2020Learning Complete 3D Morphable Face Models from Images and Videoscited by 35
5 more
  1. 2021How to Design a Three-Stage Architecture for Audio-Visual Active Speaker Detection in the Wildcited by 30
  2. 2018ExpNet: Landmark-Free, Deep, 3D Facial Expressionscited by 29
  3. 2021Look Who's Talking: Active Speaker Detection in the Wildcited by 21
  4. 2019Towards Pose-invariant Lip-Readingcited by 12
  5. 2020Real-time Facial Expression Recognition "In The Wild'' by Disentangling 3D Expression from Identitycited by 8

Built on by

Not cited yet.