mlsanityData to 7 Oct 2026

Improving End-to-end Speech Translation by Leveraging Auxiliary Speech and Text Data

arXiv 2212.01778 ↗First posted 4 Dec 2022eess.AS cs.AI cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

We present a method for introducing a text encoder into pre-trained end-to-end speech translation systems. It enhances the ability of adapting one modality (i.e., source-language speech) to another (i.e., source-language text). Thus, the speech translation model can learn from both unlabeled and labeled data, especially when the source-language text data is abundant. Beyond this, we present a denoising method to build a robust text encoder that can deal with both normal and noisy text data. Our system sets new state-of-the-arts on the MuST-C En-De, En-Fr, and LibriSpeech En-Fr tasks.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 37 of its 54 references that are in the map. First posted on arXiv 4 Dec 2022.

Builds on

37 of 54 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  6. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  7. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  8. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  9. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  10. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 239
  11. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 174
  12. 2019Consistency Regularization for Generative Adversarial Networkscited by 134
25 more
  1. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  2. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  3. 2018End-to-End Automatic Speech Translation of Audiobookscited by 65
  4. 2022Unified Speech-Text Pre-training for Speech Translation and Recognitioncited by 64
  5. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
  6. 2019End-to-End Speech Translation with Knowledge Distillationcited by 58
  7. 2018Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translationcited by 57
  8. 2020Curriculum Pre-training for End-to-End Speech Translationcited by 57
  9. 2020Bridging the Modality Gap for Speech-to-Text Translationcited by 55
  10. 2018Tied Multitask Learning for Neural Speech Translationcited by 54
  11. 2020Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translationcited by 54
  12. 2021Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Taskcited by 54
  13. 2020A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Taskscited by 51
  14. 2021Learning Shared Semantic Space for Speech-to-Text Translationcited by 49
  15. 2021Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoderscited by 48
  16. 2021Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translationcited by 47
  17. 2021End-to-end Speech Translation via Cross-modal Progressive Trainingcited by 47
  18. 2022SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-trainingcited by 40
  19. 2019Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translationcited by 39
  20. 2020Self-Training for End-to-End Speech Translationcited by 36
  21. 2020Continuous Speech Separation with Conformercited by 36
  22. 2018Augmenting Librispeech with French Translations: A Multimodal Corpus for Direct Speech Translation Evaluationcited by 34
  23. 2020Consecutive Decoding for Speech-to-text Translationcited by 28
  24. 2021End-to-End Speech Translation with Pre-trained Models and Adapters: UPC at IWSLT 2021cited by 16
  25. 2020Jointly Trained Transformers models for Spoken Language Translationcited by 10

Built on by

Cited by year

Citing papers per year: 2023: 8; 2024: 1.

9 papers.

  1. 2023CMOT: Cross-modal Mixup via Optimal Transport for Speech Translationcited by 22
  2. 2023CTC-based Non-autoregressive Speech Translationcited by 9
  3. 2023Tuning Large language model for End-to-end Speech Translationcited by 9
  4. 2023Modality Adaption or Regularization? A Case Study on End-to-End Speech Translationcited by 6
  5. 2023Rethinking and Improving Multi-task Learning for End-to-end Speech Translationcited by 6
  6. 2023Soft Alignment of Modality Space for End-to-end Speech Translationcited by 4
  7. 2023Bridging the Granularity Gap for Acoustic Modelingcited by 2
  8. 2024CrossVoice: Crosslingual Prosody Preserving Cascade-S2ST using Transfer Learningcited by 1
  9. 2023An Initialization Schema for Neuronal Networks on Tabular Datacited by 0