NAIST Simultaneous Speech Translation System for IWSLT 2024
Not on the map. Why?
Abstract (arXiv)
This paper describes NAIST's submission to the simultaneous track of the IWSLT 2024 Evaluation Campaign: English-to-{German, Japanese, Chinese} speech-to-text translation and English-to-Japanese speech-to-speech translation. We develop a multilingual end-to-end speech-to-text translation model combining two pre-trained language models, HuBERT and mBART. We trained this model with two decoding policies, Local Agreement (LA) and AlignAtt. The submitted models employ the LA policy because it outperformed the AlignAtt policy in previous models. Our speech-to-speech translation method is a cascade of the above speech-to-text model and an incremental text-to-speech (TTS) module that incorporates a phoneme estimation model, a parallel acoustic model, and a parallel WaveGAN vocoder. We improved our incremental TTS by applying the Transformer architecture with the AlignAtt policy for the estimation model. The results show that our upgraded TTS module contributed to improving the system performance.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 25 of its 42 references that are in the map. First posted on arXiv 30 Jun 2024.
Builds on
25 of 42 references.
References · 25
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
- 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
- 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
- 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
- 2020FastPitch: Parallel Text-to-speech with Pitch Predictioncited by 150
- 2019Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debatescited by 114
- 2020SimulEval: An Evaluation Toolkit for Simultaneous Translationcited by 94
- 2016Can neural machine translation do simultaneous translation?cited by 85
- 2020SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translationcited by 80
- 2020Data Augmenting Contrastive Learning of Speech Representations in the Time Domaincited by 63
13 more
- 2017JSUT corpus: free large-scale Japanese speech corpus for end-to-end speech synthesiscited by 57
- 2020CoVoST: A Diverse Multilingual Speech-To-Text Translation Corpuscited by 57
- 2022Over-Generation Cannot Be Rewarded: Length-Adaptive Average Lagging for Simultaneous Speech Translationcited by 56
- 2018Incremental Decoding and Training Methods for Simultaneous Translation in Neural Machine Translationcited by 51
- 2020Low-Latency Sequence-to-Sequence Speech Recognition and Translation by Partial Hypothesis Selectioncited by 47
- 2023AlignAtt: Using Attention-based Audio-Translation Alignments as a Guide for Simultaneous Speech Translationcited by 35
- 2022CUNI-KIT System for Simultaneous Speech Translation Task at IWSLT 2022cited by 30
- 2019Thinking Slow about Latency Evaluation for Simultaneous Machine Translationcited by 28
- 2019Re-Translation Strategies For Long Form, Simultaneous, Spoken Language Translationcited by 24
- 2022Does Simultaneous Speech Translation need Simultaneous Models?cited by 22
- 2021An Empirical Study of End-to-end Simultaneous Speech Translation Decoding Strategiescited by 14
- 2023Inter-connection: Effective Connection between Pre-trained Encoder and Decoder for Speech Translationcited by 3
- 2023Average Token Delay: A Duration-aware Latency Metric for Simultaneous Translationcited by 2
Built on by
Citing papers per year: 2024: 1.
1 paper.