mlsanityData to 14 Oct 2026

Towards a Deep Understanding of Multilingual End-to-End Speech Translation

arXiv 2310.20456 ↗First posted 31 Oct 2023cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

In this paper, we employ Singular Value Canonical Correlation Analysis (SVCCA) to analyze representations learnt in a multilingual end-to-end speech translation model trained over 22 languages. SVCCA enables us to estimate representational similarity across languages and layers, enhancing our understanding of the functionality of multilingual speech translation and its potential connection to multilingual neural machine translation. The multilingual speech translation model is trained on the CoVoST 2 dataset in all possible directions, and we utilize LASER to extract parallel bitext data for SVCCA analysis. We derive three major findings from our analysis: (I) Linguistic similarity loses its efficacy in multilingual speech translation when the training data for a specific language is limited. (II) Enhanced encoder representations and well-aligned audio-text data significantly improve translation quality, surpassing the bilingual counterparts when the training data is not compromised. (III) The encoder representations of multilingual speech translation demonstrate superior performance in predicting phonetic features in linguistic typology prediction. With these findings, we propose that releasing the constraint of limited data for low-resource languages and subsequently combining them with linguistically related high-resource languages could offer a more effective approach for multilingual end-to-end speech translation.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 55 of its 77 references that are in the map. First posted on arXiv 31 Oct 2023.

Builds on

55 of 77 references.

References · 55

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  6. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  8. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  9. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  10. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  11. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  12. 2020Beyond English-Centric Multilingual Machine Translationcited by 561
38 more
  1. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 469
  2. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  3. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 257
  4. 2019Massively Multilingual Neural Machine Translationcited by 246
  5. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 242
  6. 2020Are All Languages Created Equal in Multilingual BERT?cited by 207
  7. 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 181
  8. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 176
  9. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  10. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  11. 2022The Geometry of Multilingual Language Model Representationscited by 95
  12. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  13. 2020Speech Translation and the End-to-End Promise: Taking Stock of Where We Arecited by 77
  14. 2019Investigating Multilingual NMT Representations at Scalecited by 68
  15. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
  16. 2022STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translationcited by 57
  17. 2020Bridging the Modality Gap for Speech-to-Text Translationcited by 55
  18. 2021Lightweight Adapter Tuning for Multilingual Speech Translationcited by 54
  19. 2017Learning Joint Multilingual Sentence Representations with Neural Machine Translationcited by 52
  20. 2021Learning Shared Semantic Space for Speech-to-Text Translationcited by 49
  21. 2021Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoderscited by 48
  22. 2021Learning Language Specific Sub-network for Multilingual Machine Translationcited by 48
  23. 2019Multilingual Neural Machine Translation with Language Clusteringcited by 47
  24. 2021Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translationcited by 47
  25. 2021End-to-end Speech Translation via Cross-modal Progressive Trainingcited by 47
  26. 2017Learning Language Representations for Typology Predictioncited by 40
  27. 2021Counter-Interference Adapter for Multilingual Machine Translationcited by 40
  28. 2019Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translationcited by 39
  29. 2019Multilingual End-to-End Speech Translationcited by 37
  30. 2018Parameter Sharing Methods for Multilingual Self-Attentional Translation Modelscited by 33
  31. 2020Probing Multilingual BERT for Genetic and Typological Signalscited by 27
  32. 2022Sample, Translate, Recombine: Leveraging Audio Alignments for Data Augmentation in End-to-end Speech Translationcited by 27
  33. 2022End-to-End Speech Translation for Code Switched Speechcited by 26
  34. 2022Revisiting End-to-End Speech-to-Text Translation From Scratchcited by 26
  35. 2020Adaptive Feature Selection for End-to-End Speech Translationcited by 25
  36. 2020Bridging Linguistic Typology and Multilingual Machine Translation with Multi-View Language Representationscited by 21
  37. 2021Regularizing End-to-End Speech Translation with Triangular Decomposition Agreementcited by 16
  38. 2022Multilingual Machine Translation with Hyper-Adapterscited by 15

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 1.

3 papers.

  1. 2025Low-Rank and Sparse Model Merging for Multi-Lingual Speech Recognition and Translationcited by 6
  2. 2024Language-Independent Representations Improve Zero-Shot Summarizationcited by 0
  3. 2024Braille-to-Speech Generator: Audio Generation Based on Joint Fine-Tuning of CLIP and Fastspeech2cited by 0