Modular Speech-to-Text Translation for Zero-Shot Cross-Modal Transfer
Not on the map. Why?
Abstract (arXiv)
Recent research has shown that independently trained encoders and decoders, combined through a shared fixed-size representation, can achieve competitive performance in speech-to-text translation. In this work, we show that this type of approach can be further improved with multilingual training. We observe significant improvements in zero-shot cross-modal speech translation, even outperforming a supervised approach based on XLSR for several languages.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 21 of its 27 references that are in the map. First posted on arXiv 5 Oct 2023.
Builds on
21 of 27 references.
References · 21
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
- 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
- 2020Beyond English-Centric Multilingual Machine Translationcited by 561
- 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
- 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
- 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 178
- 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
- 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
- 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
- 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
9 more
- 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
- 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
- 2020Bridging the Modality Gap for Speech-to-Text Translationcited by 55
- 2022Bitext Mining Using Distilled Sentence Representations for Low-Resource Languagescited by 48
- 2022SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translationscited by 35
- 2022SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representationcited by 32
- 2022T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translationcited by 16
- 2017Towards speech-to-text translation without speech recognitioncited by 11
- 2020Enabling Zero-shot Multilingual Spoken Language Translation with Language-Specific Encoders and Decoderscited by 10
Built on by
Citing papers per year: 2024: 2.
2 papers.