mlsanityData to 7 Oct 2026

AV-TranSpeech: Audio-Visual Robust Speech-to-Speech Translation

arXiv 2305.15403 ↗First posted 24 May 2023cs.CL cs.SD eess.AScited by 6

Not on the map. Why?

Abstract (arXiv)

Direct speech-to-speech translation (S2ST) aims to convert speech from one language into another, and has demonstrated significant progress to date. Despite the recent success, current S2ST models still suffer from distinct degradation in noisy environments and fail to translate visual speech (i.e., the movement of lips and teeth). In this work, we present AV-TranSpeech, the first audio-visual speech-to-speech (AV-S2ST) translation model without relying on intermediate text. AV-TranSpeech complements the audio stream with visual information to promote system robustness and opens up a host of practical applications: dictation or dubbing archival films. To mitigate the data scarcity with limited parallel AV-S2ST data, we 1) explore self-supervised pre-training with unlabeled audio-visual data to learn contextual representation, and 2) introduce cross-modal distillation with S2ST models trained on the audio-only corpus to further reduce the requirements of visual data. Experimental results on two language pairs demonstrate that AV-TranSpeech outperforms audio-only models under all settings regardless of the type of noise. With low-resource audio-visual data (10h, 30h), cross-modal distillation yields an improvement of 7.6 BLEU on average compared with baselines. Audio samples are available at https://AV-TranSpeech.github.io

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 38 of its 50 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

38 of 50 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  5. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  6. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  7. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  8. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  9. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  10. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 400
  11. 2018Deep Audio-Visual Speech Recognitioncited by 318
  12. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 301
26 more
  1. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
  2. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  3. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  4. 2021Direct speech-to-speech translation with discrete unitscited by 148
  5. 2022ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speechcited by 147
  6. 2022FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesiscited by 136
  7. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  8. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  9. 2020Lipreading using Temporal Convolutional Networkscited by 98
  10. 2019ASR is all you need: cross-modal distillation for lip readingcited by 85
  11. 2022CVSS Corpus and Massively Multilingual Speech-to-Speech Translationcited by 71
  12. 2017Combining Residual Networks with LSTMs for Lipreadingcited by 66
  13. 2021Sub-word Level Lip Reading With Visual Attentioncited by 60
  14. 2021Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpuscited by 60
  15. 2020UWSpeech: Speech to Speech Translation for Unwritten Languagescited by 43
  16. 2022Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentationcited by 43
  17. 2021Bilateral Denoising Diffusion Modelscited by 36
  18. 2022SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translationscited by 35
  19. 2022TranSpeech: Speech-to-Speech Translation With Bilateral Perturbationcited by 34
  20. 2019Speech-to-speech Translation between Untranscribed Unknown Languagescited by 27
  21. 2021SingGAN: Generative Adversarial Network For High-Fidelity Singing Voice Generationcited by 27
  22. 2022Speech-to-Speech Translation For A Real-world Unwritten Languagecited by 25
  23. 2022Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translationcited by 20
  24. 2020Transfer Learning from Speech Synthesis to Voice Conversion with Non-Parallel Training Datacited by 17
  25. 2020From Speaker Verification to Multispeaker Speech Synthesis, Deep Transfer with Feedback Constraintcited by 14
  26. 2022Learning Lip-Based Audio-Visual Speaker Embeddings with AV-HuBERTcited by 8

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 3; 2025: 1.

6 papers.

  1. 2024Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesiscited by 97
  2. 2023AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representationcited by 14
  3. 2023TransFace: Unit-Based Audio-Visual Speech Synthesizer for Talking Head Translationcited by 4
  4. 2024AnnoTheia: A Semi-Automatic Annotation Toolkit for Audio-Visual Speech Technologiescited by 1
  5. 2024Improving Lip-synchrony in Direct Audio-Visual Speech-to-Speech Translationcited by 0
  6. 2025MAVFlow: Preserving Paralinguistic Elements with Conditional Flow Matching for Zero-Shot AV2AV Multilingual Translationcited by 0