mlsanityData to 7 Oct 2026

Lip2AudSpec: Speech reconstruction from silent lip movements video

arXiv 1710.09798 ↗First posted 26 Oct 2017cs.CV eess.AS eess.IVcited by 32

Not on the map. Why?

Abstract (arXiv)

In this study, we propose a deep neural network for reconstructing intelligible speech from silent lip movement videos. We use auditory spectrogram as spectral representation of speech and its corresponding sound generation method resulting in a more natural sounding reconstructed speech. Our proposed network consists of an autoencoder to extract bottleneck features from the auditory spectrogram which is then used as target to our main lip reading network comprising of CNN, LSTM and fully connected layers. Our experiments show that the autoencoder is able to reconstruct the original auditory spectrogram with a 98% correlation and also improves the quality of reconstructed speech from the main lip reading network. Our model, trained jointly on different speakers is able to extract individual speaker characteristics and gives promising results of reconstructing intelligible speech with superior word recognition accuracy.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Oct 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 13; 2023: 10; 2024: 1; 2025: 1; 2026: 1.

32 papers.

Sort:

By area:

  1. 2022Learning in Audio-visual Context: A Review, Analysis, and New Perspectivecited by 44
  2. 2022Lip to Speech Synthesis with Visual Context Attentional GANcited by 41
  3. 2022Distinguishing Homophenes Using Multi-Head Visual-Audio Memory for Lip Readingcited by 39
  4. 2022SVTS: Scalable Video-to-Speech Synthesiscited by 35
  5. 2021End-to-End Video-To-Speech Synthesis using Generative Adversarial Networkscited by 27
  6. 2023Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applicationscited by 27
  7. 2023DiffV2S: Diffusion-based Video-to-Speech Synthesis with Vision-guided Speaker Embeddingcited by 22
  8. 2023Lip-to-Speech Synthesis in the Wild with Multi-task Learningcited by 21
  9. 2022Multi-modality Associative Bridging through Memory: Speech Sound Recollected from Face Videocited by 18
  10. 2023AKVSR: Audio Knowledge Empowered Visual Speech Recognition by Compressing Audio Knowledge of a Pretrained Modelcited by 16
  11. 2021LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Readingcited by 10
  12. 2022FastLTS: Non-Autoregressive End-to-End Unconstrained Lip-to-Speech Synthesiscited by 8
20 more
  1. 2022Lip-to-Speech Synthesis for Arbitrary Speakers in the Wildcited by 8
  2. 2024Towards Accurate Lip-to-Speech Synthesis in-the-Wildcited by 7
  3. 2022VCVTS: Multi-speaker Video-to-Speech synthesis via cross-modal knowledge transfer from voice conversioncited by 6
  4. 2022VisageSynTalk: Unseen Speaker Video-to-Speech Synthesis via Speech-Visage Feature Selectioncited by 6
  5. 20213D Convolutional Neural Networks for Ultrasound-Based Silent Speech Interfacescited by 4
  6. 2021Facetron: A Multi-speaker Face-to-Speech Model based on Cross-modal Latent Representationscited by 4
  7. 2022Learning Speaker-specific Lip-to-Speech Generationcited by 3
  8. 2023Large-scale unsupervised audio pre-training for video-to-speech synthesiscited by 3
  9. 2023Speech Audio Synthesis from Tagged MRI and Non-Negative Matrix Factorization via Plastic Transformercited by 3
  10. 2021Towards a practical lip-to-speech conversion system using deep neural networks and mobile application frontendcited by 2
  11. 2021Improving Ultrasound Tongue Image Reconstruction from Lip Images Using Self-supervised Learning and Attention Mechanismcited by 2
  12. 2022Tagged-MRI Sequence to Audio Synthesis via Self Residual Attention Guided Heterogeneous Translatorcited by 1
  13. 2022EPG2S: Speech Generation and Speech Enhancement based on Electropalatography and Audio Signals using Multimodal Learningcited by 1
  14. 2022Dual-Path Cross-Modal Attention for better Audio-Visual Speech Extractioncited by 1
  15. 2023Synthesizing audio from tongue motion during speech using tagged MRI via transformercited by 1
  16. 2023On the Audio-visual Synchronization for Lip-to-Speech Synthesiscited by 1
  17. 2023RobustL2S: Speaker-Specific Lip-to-Speech Synthesis exploiting Self-Supervised Representationscited by 1
  18. 2023Audio-visual video-to-speech synthesis with synthesized input audiocited by 1
  19. 2025Speech Audio Generation from dynamic MRI via a Knowledge Enhanced Conditional Variational Autoencodercited by 0
  20. 2026Cross-Modal Masking for Robust Silent Speech Synthesis Using sEMG and Lipreadingcited by 0