mlsanityData to 7 Oct 2026

Learning Shared Semantic Space for Speech-to-Text Translation

arXiv 2105.03095 ↗First posted 7 May 2021cs.CLcited by 49

Not on the map. Why?

Abstract (arXiv)

Having numerous potential applications and great impact, end-to-end speech translation (ST) has long been treated as an independent task, failing to fully draw strength from the rapid advances of its sibling - text machine translation (MT). With text and audio inputs represented differently, the modality gap has rendered MT data and its end-to-end models incompatible with their ST counterparts. In observation of this obstacle, we propose to bridge this representation gap with Chimera. By projecting audio and text features to a common semantic representation, Chimera unifies MT and ST tasks and boosts the performance on ST benchmarks, MuST-C and Augmented Librispeech, to a new state-of-the-art. Specifically, Chimera obtains 27.1 BLEU on MuST-C EN-DE, improving the SOTA by a +1.9 BLEU margin. Further experimental analyses demonstrate that the shared semantic space indeed conveys common knowledge between these two tasks and thus paves a new way for augmenting training resources across modalities. Code, data, and resources are available at https://github.com/Glaciohound/Chimera-ST.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Builds on 39 of its 64 references that are in the map. First posted on arXiv 7 May 2021.

Builds on

39 of 64 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  4. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  5. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  6. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  7. 2020ESPnet-ST: All-in-One Speech Translation Toolkitcited by 82
  8. 2018Pre-training on high-resource speech recognition improves low-resource speech-to-text translationcited by 70
  9. 2018End-to-End Automatic Speech Translation of Audiobookscited by 65
  10. 2019Improving Multilingual Sentence Embedding using Bi-directional Dual Encoder with Additive Margin Softmaxcited by 58
  11. 2019End-to-End Speech Translation with Knowledge Distillationcited by 58
  12. 2018Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translationcited by 57
27 more
  1. 2020Curriculum Pre-training for End-to-End Speech Translationcited by 57
  2. 2020Bridging the Modality Gap for Speech-to-Text Translationcited by 55
  3. 2020Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translationcited by 54
  4. 2020A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Taskscited by 51
  5. 2019A Comparative Study on End-to-end Speech to Text Translationcited by 40
  6. 2019Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translationcited by 39
  7. 2019Multilingual End-to-End Speech Translationcited by 37
  8. 2020Self-Training for End-to-End Speech Translationcited by 36
  9. 2018Towards Robust Neural Machine Translationcited by 35
  10. 2018Augmenting Librispeech with French Translations: A Multimodal Corpus for Direct Speech Translation Evaluationcited by 34
  11. 2018A neural interlingua for multilingual machine translationcited by 29
  12. 2020Consecutive Decoding for Speech-to-text Translationcited by 28
  13. 2019Analyzing ASR pretraining for low-resource speech-to-text translationcited by 27
  14. 2020Adaptive Feature Selection for End-to-End Speech Translationcited by 25
  15. 2019Fluent Translations from Disfluent Speech in End-to-End Speech Translationcited by 22
  16. 2017Neural Lattice-to-Sequence Models for Uncertain Inputscited by 20
  17. 2018Multilingual NMT with a language-independent attention bridgecited by 20
  18. 2019On Using SpecAugment for End-to-End Speech Translationcited by 19
  19. 2019Lattice Transformer for Speech Translationcited by 15
  20. 2019Self-Attentional Models for Lattice Inputscited by 12
  21. 2019Data Efficient Direct Speech-to-Text Translation with Modality Agnostic Meta-Learningcited by 11
  22. 2020Jointly Trained Transformers models for Spoken Language Translationcited by 10
  23. 2018Towards Fluent Translations from Disfluent Speechcited by 7
  24. 2019Breaking the Data Barrier: Towards Robust Speech Translation via Adversarial Stability Trainingcited by 7
  25. 2018The USTC-NEL Speech Translation system at IWSLT 2018cited by 1
  26. 2019Instance-Based Model Adaptation For Direct Speech Translationcited by 1
  27. 2020"Listen, Understand and Translate": Triple Supervision Decouples End-to-end Speech-to-text Translationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 16; 2023: 19; 2024: 4; 2025: 4; 2026: 2.

49 papers.

Sort:

  1. 2021A Survey of Transformerscited by 399
  2. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
  3. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
  4. 2022STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translationcited by 57
  5. 2023AlignAtt: Using Attention-based Audio-Translation Alignments as a Guide for Simultaneous Speech Translationcited by 35
  6. 2021Learning When to Translate for Streaming Speechcited by 30
  7. 2022Revisiting End-to-End Speech-to-Text Translation From Scratchcited by 26
  8. 2023Pre-training for Speech Translation: CTC Meets Optimal Transportcited by 25
  9. 2023CMOT: Cross-modal Mixup via Optimal Transport for Speech Translationcited by 22
  10. 2022M3ST: Mix at Three Levels for Speech Translationcited by 21
  11. 2022WACO: Word-Aligned Contrastive Learning for Speech Translationcited by 21
  12. 2024LoFormer: Local Frequency Transformer for Image Deblurringcited by 19
37 more
  1. 2023DUB: Discrete Unit Back-translation for Speech Translationcited by 18
  2. 2021Regularizing End-to-End Speech Translation with Triangular Decomposition Agreementcited by 16
  3. 2022M-Adapter: Modality Adaptation for End-to-End Speech-to-Text Translationcited by 15
  4. 2023Understanding and Bridging the Modality Gap for Speech Translationcited by 15
  5. 2023Recent Advances in Direct Speech-to-text Translationcited by 15
  6. 2022Efficient yet Competitive Speech Translation: FBK@IWSLT2022cited by 14
  7. 2024Pushing the Limits of Zero-shot End-to-End Speech Translationcited by 14
  8. 2022Non-Parametric Domain Adaptation for End-to-End Speech Translationcited by 12
  9. 2022MMSpeech: Multi-modal Multi-task Encoder-Decoder Pre-training for Speech Recognitioncited by 9
  10. 2022Improving End-to-end Speech Translation by Leveraging Auxiliary Speech and Text Datacited by 9
  11. 2023Back Translation for Speech-to-text Translation Without Transcriptscited by 9
  12. 2023Tuning Large language model for End-to-end Speech Translationcited by 9
  13. 2023End-to-End Speech-to-Text Translation: A Surveycited by 9
  14. 2025Bridging Domain Generalization to Multimodal Domain Generalization via Unified Representationscited by 9
  15. 2022Label Anchored Contrastive Learning for Language Understandingcited by 7
  16. 2023Improving Speech Translation by Cross-Modal Multi-Grained Contrastive Learningcited by 7
  17. 2022On the Impact of Noises in Crowd-Sourced Data for Speech Translationcited by 6
  18. 2022Discrete Cross-Modal Alignment Enables Zero-Shot Speech Translationcited by 6
  19. 2022SegAugment: Maximizing the Utility of Speech Translation Data with Segmentation-based Augmentationscited by 6
  20. 2023Adapting Offline Speech Translation Models for Streaming with Future-Aware Distillation and Inferencecited by 6
  21. 2023An Empirical Study of Consistency Regularization for End-to-End Speech-to-Text Translationcited by 6
  22. 2023Rethinking and Improving Multi-task Learning for End-to-end Speech Translationcited by 6
  23. 2022An Analysis of Semantically-Aligned Speech-Text Embeddingscited by 5
  24. 2024CoSTA: Code-Switched Speech Translation using Aligned Speech-Text Interleavingcited by 4
  25. 2022AdaTranS: Adapting with Boundary-based Shrinking for End-to-End Speech Translationcited by 3
  26. 2023Decouple Non-parametric Knowledge Distillation For End-to-end Speech Translationcited by 3
  27. 2023Improving speech translation by fusing speech and textcited by 3
  28. 2023Towards a Deep Understanding of Multilingual End-to-End Speech Translationcited by 3
  29. 2024Representation Purification for End-to-End Speech Translationcited by 3
  30. 2025DTW-Align: Bridging the Modality Gap in End-to-End Speech Translation with Dynamic Time Warping Alignmentcited by 2
  31. 2023Efficient CTC Regularization via Coarse Labels for End-to-End Speech Translationcited by 1
  32. 2023Speech Translation with Foundation Models and Optimal Transport: UPC at IWSLT23cited by 1
  33. 2023Fast Training of NMT Model with Data Sortingcited by 0
  34. 2025Continual Cross-Modal Generalizationcited by 0
  35. 2025Optimal Multi-Task Learning at Regularization Horizon for Speech Translation Taskcited by 0
  36. 2026Cross-Modal-Domain Generalization Through Semantically Aligned Discrete Representationscited by 0
  37. 2026Generic Interpretation Approach for Transformer Models Incorporating Heterogenous Attention Structurescited by 0