mlsanityData to 14 Oct 2026

Soft Alignment of Modality Space for End-to-end Speech Translation

arXiv 2312.10952 ↗First posted 18 Dec 2023cs.CL cs.AI cs.SDcited by 4

Not on the map. Why?

Abstract (arXiv)

End-to-end Speech Translation (ST) aims to convert speech into target text within a unified model. The inherent differences between speech and text modalities often impede effective cross-modal and cross-lingual transfer. Existing methods typically employ hard alignment (H-Align) of individual speech and text segments, which can degrade textual representations. To address this, we introduce Soft Alignment (S-Align), using adversarial training to align the representation spaces of both modalities. S-Align creates a modality-invariant space while preserving individual modality quality. Experiments on three languages from the MuST-C dataset show S-Align outperforms H-Align across multiple tasks and offers translation capabilities on par with specialized translation models.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 23 of its 31 references that are in the map. First posted on arXiv 18 Dec 2023.

Builds on

23 of 31 references.

References · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Generative Adversarial Networkscited by 5,627
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2015Domain-Adversarial Training of Neural Networkscited by 3,044
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  7. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  8. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  9. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 176
  10. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  11. 2021Intermediate Loss Regularization for CTC-based Speech Recognitioncited by 96
  12. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
11 more
  1. 2022STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translationcited by 57
  2. 2021Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Taskcited by 54
  3. 2021Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoderscited by 48
  4. 2021Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translationcited by 47
  5. 2021End-to-end Speech Translation via Cross-modal Progressive Trainingcited by 47
  6. 2022SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-trainingcited by 40
  7. 2023Pre-training for Speech Translation: CTC Meets Optimal Transportcited by 25
  8. 2022WACO: Word-Aligned Contrastive Learning for Speech Translationcited by 21
  9. 2022Improving End-to-end Speech Translation by Leveraging Auxiliary Speech and Text Datacited by 9
  10. 2023Rethinking and Improving Multi-task Learning for End-to-end Speech Translationcited by 6
  11. 2023Bridging the Granularity Gap for Acoustic Modelingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2024Representation Purification for End-to-End Speech Translationcited by 3
  2. 2025DTW-Align: Bridging the Modality Gap in End-to-End Speech Translation with Dynamic Time Warping Alignmentcited by 2
  3. 2025Optimizing Speech Multi-View Feature Fusion through Conditional Computationcited by 0
  4. 2026Cross-Modal Robustness Transfer (CMRT): Training Robust Speech Translation Models Using Adversarial Textcited by 0