mlsanityData to 7 Oct 2026

LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Reading

arXiv 2112.04748 ↗First posted 9 Dec 2021cs.SD cs.AI eess.AScited by 10

Not on the map. Why?

Abstract (arXiv)

The aim of this work is to investigate the impact of crossmodal self-supervised pre-training for speech reconstruction (video-to-audio) by leveraging the natural co-occurrence of audio and visual streams in videos. We propose LipSound2 which consists of an encoder-decoder architecture and location-aware attention mechanism to map face image sequences to mel-scale spectrograms directly without requiring any human annotations. The proposed LipSound2 model is firstly pre-trained on ~2400h multi-lingual (e.g. English and German) audio-visual data (VoxCeleb2). To verify the generalizability of the proposed method, we then fine-tune the pre-trained model on domain-specific datasets (GRID, TCD-TIMIT) for English speech reconstruction and achieve a significant improvement on speech quality and intelligibility compared to previous approaches in speaker-dependent and -independent settings. In addition to English, we conduct Chinese speech reconstruction on the CMLR dataset to verify the impact on transferability. Lastly, we train the cascaded lip reading (video-to-text) system by fine-tuning the generated audios on a pre-trained speech recognition system and achieve state-of-the-art performance on both English and Chinese benchmark datasets.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 54 of its 101 references that are in the map. First posted on arXiv 9 Dec 2021.

Builds on

54 of 101 references.

References · 54

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  3. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  4. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  5. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  6. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  7. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
  8. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  9. 2016Colorful Image Colorizationcited by 1,113
  10. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  11. 2016Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networkscited by 895
  12. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
38 more
  1. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  2. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  3. 2017Look, Listen and Learncited by 452
  4. 2015Attention-Based Models for Speech Recognitioncited by 400
  5. 2018Deep Audio-Visual Speech Recognitioncited by 318
  6. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 308
  7. 2017Objects that Soundcited by 254
  8. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 246
  9. 2016Lip Reading Sentences in the Wildcited by 226
  10. 2018Tracking Emerges by Colorizing Videoscited by 166
  11. 2018AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scalecited by 125
  12. 2020Lipreading using Temporal Convolutional Networkscited by 98
  13. 2020Learning Representations from Audio-Visual Spatial Alignmentcited by 95
  14. 2016LipNet: End-to-End Sentence-level Lipreadingcited by 79
  15. 2019Jasper: An End-to-End Convolutional Neural Acoustic Modelcited by 75
  16. 2020Learning Individual Speaking Styles for Accurate Lip to Speech Synthesiscited by 74
  17. 2017Combining Residual Networks with LSTMs for Lipreadingcited by 66
  18. 2021Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videoscited by 62
  19. 2018Perfect match: Improved cross-modal embeddings for audio-visual synchronisationcited by 60
  20. 2018Large-Scale Visual Speech Recognitioncited by 57
  21. 2017Vid2speech: Speech Reconstruction from Silent Videocited by 53
  22. 2018LRW-1000: A Naturally-Distributed Large-Scale Benchmark for Lip Reading in the Wildcited by 51
  23. 2019Hearing Lips: Improving Lip Reading by Distilling Speech Recognizerscited by 39
  24. 2017Lip2AudSpec: Speech reconstruction from silent lip movements videocited by 32
  25. 2017Improved Speech Reconstruction from Silent Videocited by 26
  26. 2018Deep Lip Reading: a comparison of models and an online applicationcited by 26
  27. 2019Video-Driven Speech Reconstruction using Generative Adversarial Networkscited by 25
  28. 2019A Cascade Sequence-to-Sequence Model for Chinese Mandarin Lip Readingcited by 25
  29. 2020FaceFilter: Audio-visual speech separation using still imagescited by 25
  30. 2019Learning Spatio-Temporal Features with Two-Stream Deep 3D CNNs for Lipreadingcited by 22
  31. 2020Vocoder-Based Speech Synthesis from Silent Videoscited by 20
  32. 2019Lipper: Synthesizing Thy Speech using Multi-View Lipreadingcited by 19
  33. 2016Lipreading with Long Short-Term Memorycited by 14
  34. 2018LCANet: End-to-End Lipreading with Cascaded Attention-CTCcited by 14
  35. 2020Pseudo-Convolutional Policy Gradient for Sequence-to-Sequence Lip-Readingcited by 12
  36. 2020DualLip: A System for Joint Lip Reading and Generationcited by 10
  37. 2017Visual Features for Context-Aware Speech Recognitioncited by 8
  38. 2017Improving Speaker-Independent Lipreading with Domain-Adversarial Trainingcited by 7

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 6; 2024: 0; 2025: 3.

10 papers.

  1. 2023HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesiscited by 35
  2. 2022Disentangling Prosody Representations with Unsupervised Speech Reconstructioncited by 8
  3. 2023Hearing Lips in Noise: Universal Viseme-Phoneme Mapping and Transfer for Robust Audio-Visual Speech Recognitioncited by 7
  4. 2025Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representationcited by 5
  5. 2025MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognitioncited by 4
  6. 2023Large-scale unsupervised audio pre-training for video-to-speech synthesiscited by 3
  7. 2025NaturalL2S: End-to-End High-quality Multispeaker Lip-to-Speech Synthesis with Differential Digital Signal Processingcited by 3
  8. 2023Looking and Listening: Audio Guided Text Recognitioncited by 1
  9. 2023RobustL2S: Speaker-Specific Lip-to-Speech Synthesis exploiting Self-Supervised Representationscited by 1
  10. 2023Audio-visual video-to-speech synthesis with synthesized input audiocited by 1