Improving speech translation by fusing speech and text
Not on the map. Why?
Abstract (arXiv)
In speech translation, leveraging multimodal data to improve model performance and address limitations of individual modalities has shown significant effectiveness. In this paper, we harness the complementary strengths of speech and text, which are disparate modalities. We observe three levels of modality gap between them, denoted by Modal input representation, Modal semantic, and Modal hidden states. To tackle these gaps, we propose Fuse-Speech-Text (FST), a cross-modal model which supports three distinct input modalities for translation: speech, text, and fused speech-text. We leverage multiple techniques for cross-modal alignment and conduct a comprehensive analysis to assess its impact on speech translation, machine translation, and fused speech-text translation. We evaluate FST on MuST-C, GigaST, and newstest benchmark. Experiments show that the proposed FST achieves an average 34.0 BLEU on MuST-C EnrightarrowDe/Es/Fr (vs SOTA +1.1 BLEU). Further experiments demonstrate that FST does not degrade on MT task, as observed in prior works. Instead, it yields an average improvement of 3.2 BLEU over the pre-trained MT model.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 43 of its 60 references that are in the map. First posted on arXiv 23 May 2023.
Builds on
43 of 60 references.
References · 43
- 2017Attention Is All You Needcited by 59,932
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
- 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
- 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
- 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
- 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
- 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
- 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
- 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
- 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
- 2018Pre-training on high-resource speech recognition improves low-resource speech-to-text translationcited by 70
31 more
- 2018End-to-End Automatic Speech Translation of Audiobookscited by 65
- 2022Unified Speech-Text Pre-training for Speech Translation and Recognitioncited by 64
- 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
- 2019End-to-End Speech Translation with Knowledge Distillationcited by 58
- 2018Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translationcited by 57
- 2020Curriculum Pre-training for End-to-End Speech Translationcited by 57
- 2022STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translationcited by 57
- 2020Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translationcited by 54
- 2021Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Taskcited by 54
- 2020A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Taskscited by 51
- 2021Learning Shared Semantic Space for Speech-to-Text Translationcited by 49
- 2021Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoderscited by 48
- 2021Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translationcited by 47
- 2021End-to-end Speech Translation via Cross-modal Progressive Trainingcited by 47
- 2019Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translationcited by 39
- 2020Self-Training for End-to-End Speech Translationcited by 36
- 2020End-to-End Speech-Translation with Knowledge Distillation: FBK@IWSLT2020cited by 31
- 2022On Vision Features in Multimodal Machine Translationcited by 31
- 2021Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translationcited by 27
- 2019Fluent Translations from Disfluent Speech in End-to-End Speech Translationcited by 22
- 2017Neural Lattice-to-Sequence Models for Uncertain Inputscited by 20
- 2022GigaST: A 10,000-hour Pseudo Speech Translation Corpuscited by 20
- 2019On Using SpecAugment for End-to-End Speech Translationcited by 19
- 2019Lattice Transformer for Speech Translationcited by 15
- 2019Robust Neural Machine Translation for Clean and Noisy Speech Transcriptscited by 13
- 2019Self-Attentional Models for Lattice Inputscited by 12
- 2018Structured-based Curriculum Learning for End-to-end English-Japanese Speech Translationcited by 11
- 2019Data Efficient Direct Speech-to-Text Translation with Modality Agnostic Meta-Learningcited by 11
- 2020Jointly Trained Transformers models for Spoken Language Translationcited by 10
- 2020Cascaded Models With Cyclic Feedback For Direct Speech Translationcited by 9
- 2019Breaking the Data Barrier: Towards Robust Speech Translation via Adversarial Stability Trainingcited by 7
Built on by
Citing papers per year: 2024: 1; 2025: 2.
3 papers.