mlsanityData to 14 Oct 2026

Intelligible Lip-to-Speech Synthesis with Speech Units

arXiv 2305.19603 ↗First posted 31 May 2023cs.SD cs.CV eess.AScited by 28

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose a novel Lip-to-Speech synthesis (L2S) framework, for synthesizing intelligible speech from a silent lip movement video. Specifically, to complement the insufficient supervisory signal of the previous L2S model, we propose to use quantized self-supervised speech representations, named speech units, as an additional prediction target for the L2S model. Therefore, the proposed L2S model is trained to generate multiple targets, mel-spectrogram and speech units. As the speech units are discrete while mel-spectrogram is continuous, the proposed multi-target L2S model can be trained with strong content supervision, without using text-labeled data. Moreover, to accurately convert the synthesized mel-spectrogram into a waveform, we introduce a multi-input vocoder that can generate a clear waveform even from blurry and noisy mel-spectrogram by referring to the speech units. Extensive experimental results confirm the effectiveness of the proposed method in L2S.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 25 of its 33 references that are in the map. First posted on arXiv 31 May 2023.

Builds on

25 of 33 references.

References · 25

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  4. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  5. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  6. 2018VoxCeleb2: Deep Speaker Recognitioncited by 875
  7. 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 340
  8. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  9. 2018Deep Audio-Visual Speech Recognitioncited by 322
  10. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 308
  11. 2017Generalized End-to-End Loss for Speaker Verificationcited by 293
  12. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 287
13 more
  1. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  2. 2016Lip Reading Sentences in the Wildcited by 228
  3. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 151
  4. 2022Generative Spoken Dialogue Language Modelingcited by 126
  5. 2020Lipreading using Temporal Convolutional Networkscited by 102
  6. 2022Robust Self-Supervised Audio-Visual Speech Recognitioncited by 75
  7. 2020Learning Individual Speaking Styles for Accurate Lip to Speech Synthesiscited by 74
  8. 2021Sub-word Level Lip Reading With Visual Attentioncited by 63
  9. 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
  10. 2022Lip to Speech Synthesis with Visual Context Attentional GANcited by 41
  11. 2022Distinguishing Homophenes Using Multi-Head Visual-Audio Memory for Lip Readingcited by 39
  12. 2022SVTS: Scalable Video-to-Speech Synthesiscited by 35
  13. 2023Lip-to-Speech Synthesis in the Wild with Multi-task Learningcited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 3; 2025: 13; 2026: 5.

28 papers.

Sort:

By area:

  1. 2023DiffV2S: Diffusion-based Video-to-Speech Synthesis with Vision-guided Speaker Embeddingcited by 22
  2. 2023Lip Reading for Low-resource Languages by Learning and Combining General Speech Knowledge and Language-specific Knowledgecited by 16
  3. 2023AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representationcited by 14
  4. 2023Visual Speech Recognition for Languages with Limited Labeled Data using Automatic Labels from Whispercited by 13
  5. 2025AudioGen-Omni: A Unified Multimodal Diffusion Transformer for Video-Synchronized Audio, Speech, and Song Generationcited by 12
  6. 2023Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translationcited by 11
  7. 2025VoiceCraft-Dub: Automated Video Dubbing with Neural Codec Language Modelscited by 11
  8. 2025From Faces to Voices: Learning Hierarchical Representations for High-quality Video-to-Speechcited by 9
  9. 2024Efficient Training for Multilingual Visual Speech Recognition: Pre-training with Discretized Visual Speech Representationcited by 8
  10. 2024Separate in the Speech Chain: Cross-Modal Conditional Audio-Visual Target Speech Extractioncited by 7
  11. 2025AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generationcited by 7
  12. 2024V2SFlow: Video-to-Speech Generation with Speech Decomposition and Rectified Flowcited by 5
16 more
  1. 2025Supervising 3D Talking Head Avatars with Analysis-by-Audio-Synthesiscited by 5
  2. 2023Towards Practical and Efficient Image-to-Speech Captioning with Vision-Language Pre-training and Multi-modal Tokenscited by 4
  3. 2025NaturalL2S: End-to-End High-quality Multispeaker Lip-to-Speech Synthesis with Differential Digital Signal Processingcited by 3
  4. 2025LipDiffuser: Lip-to-Speech Generation with Conditional Diffusion Modelscited by 2
  5. 2026DiFlowDubber: Discrete Flow Matching for Automated Video Dubbing via Cross-Modal Alignment and Synchronizationcited by 2
  6. 2023RobustL2S: Speaker-Specific Lip-to-Speech Synthesis exploiting Self-Supervised Representationscited by 1
  7. 2025DiVISe: Direct Visual-Input Speech Synthesis Preserving Speaker Characteristics And Intelligibilitycited by 1
  8. 2025RESOUND: Speech Reconstruction from Silent Videos via Acoustic-Semantic Decomposed Modelingcited by 1
  9. 2025LTA-L2S: Lexical Tone-Aware Lip-to-Speech Synthesis for Mandarin with Cross-Lingual Transfer Learningcited by 1
  10. 2026The Role of Prosodic and Lexical Cues in Turn-Taking with Self-Supervised Speech Representationscited by 1
  11. 2026CoSyncDiT: Cognitive Synchronous Diffusion Transformer for Movie Dubbingcited by 1
  12. 2025VisualSpeech: Enhancing Prosody Modeling in TTS Using Videocited by 0
  13. 2025UniCUE: Unified Recognition and Generation Framework for Chinese Cued Speech Video-to-Speech Generationcited by 0
  14. 2025MuteSwap: Visual-informed Silent Video Identity Conversioncited by 0
  15. 2026SLD-L2S: Hierarchical Subspace Latent Diffusion for High-Fidelity Lip to Speech Synthesiscited by 0
  16. 2026Watch Your Speech: Text-aware Video-to-Speech Synthesis with Textual Conditioningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022VCVTS: Multi-speaker Video-to-Speech synthesis via cross-modal knowledge transfer from voice conversion
  2. 2023LipVoicer: Generating Speech from Silent Videos Guided by Lip Reading
  3. 2025Target Speaker Lipreading by Audio-Visual Self-Distillation Pretraining and Speaker Adaptation
  4. 2023High-Fidelity Speech Synthesis with Minimal Supervision: All Using Diffusion Models
  5. 2023An Experimental Study: Assessing the Combined Framework of WavLM and BEST-RQ for Text-to-Speech Synthesis
  6. 2025MELA-TTS: Joint transformer-diffusion model with representation alignment for speech synthesis
  7. 2022DelightfulTTS 2: End-to-End Speech Synthesis with Adversarial Vector-Quantized Auto-Encoders
  8. 2026Cross-Modal Masking for Robust Silent Speech Synthesis Using sEMG and Lipreading