mlsanityData to 7 Oct 2026

M-Adapter: Modality Adaptation for End-to-End Speech-to-Text Translation

arXiv 2207.00952 ↗First posted 3 Jul 2022cs.CL cs.LG cs.SDcited by 15

Not on the map. Why?

Abstract (arXiv)

End-to-end speech-to-text translation models are often initialized with pre-trained speech encoder and pre-trained text decoder. This leads to a significant training gap between pre-training and fine-tuning, largely due to the modality differences between speech outputs from the encoder and text inputs to the decoder. In this work, we aim to bridge the modality gap between speech and text to improve translation quality. We propose M-Adapter, a novel Transformer-based module, to adapt speech representations to text. While shrinking the speech sequence, M-Adapter produces features desired for speech-to-text translation via modelling global and local dependencies of a speech sequence. Our experimental results show that our model outperforms a strong baseline by up to 1 BLEU score on the Must-C EnrightarrowDE dataset.footnoteOur code is available at https://github.com/mingzi151/w2v2-st.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 25 of its 31 references that are in the map. First posted on arXiv 3 Jul 2022.

Builds on

25 of 31 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  5. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  6. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  7. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  8. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 174
  9. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 127
  10. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  11. 2019Unsupervised speech representation learning using WaveNet autoencoderscited by 90
  12. 2020Speech Translation and the End-to-End Promise: Taking Stock of Where We Arecited by 77
13 more
  1. 2020DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantizationcited by 77
  2. 2020Bridging the Modality Gap for Speech-to-Text Translationcited by 55
  3. 2021Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Taskcited by 54
  4. 2021Learning Shared Semantic Space for Speech-to-Text Translationcited by 49
  5. 2021Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoderscited by 48
  6. 2021Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translationcited by 47
  7. 2021End-to-end Speech Translation via Cross-modal Progressive Trainingcited by 47
  8. 2021CTC-based Compression for Direct Speech Translationcited by 45
  9. 2019Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translationcited by 39
  10. 2020Adaptive Feature Selection for End-to-End Speech Translationcited by 25
  11. 2021End-to-End Speech Translation with Pre-trained Models and Adapters: UPC at IWSLT 2021cited by 16
  12. 2020Phone Features Improve Speech Translationcited by 11
  13. 2021Multilingual Translation via Grafting Pre-trained Language Modelscited by 10

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 4; 2024: 7; 2025: 1.

15 papers.

Sort:

By area:

  1. 2023Investigating Pre-trained Audio Encoders in the Low-Resource Conditioncited by 11
  2. 2022Towards Relation Extraction From Speechcited by 10
  3. 2023End-to-End Speech-to-Text Translation: A Surveycited by 9
  4. 2024How do Multimodal Foundation Models Encode Text and Speech? An Analysis of Cross-Lingual and Cross-Modal Representationscited by 8
  5. 2023CoLLD: Contrastive Layer-to-layer Distillation for Compressing Multilingual Pre-trained Speech Encoderscited by 7
  6. 2022Generating Synthetic Speech from SpokenVocab for Speech Translationcited by 5
  7. 2024Pronunciation Assessment with Multi-modal Large Language Modelscited by 4
  8. 2024Towards Event Extraction from Speech with Contextual Cluescited by 3
  9. 2025Different Speech Translation Models Encode and Translate Speaker Gender Differentlycited by 3
  10. 2022RedApt: An Adaptor for wav2vec 2 Encoding Faster and Smaller Speech Translation without Quality Compromisecited by 2
  11. 2023Speech Translation with Foundation Models and Optimal Transport: UPC at IWSLT23cited by 1
  12. 2024SimulSeamless: FBK at IWSLT 2024 Simultaneous Speech Translationcited by 1
3 more
  1. 2024Leveraging Parameter-Efficient Transfer Learning for Multi-Lingual Text-to-Speech Adaptationcited by 1
  2. 2024CTC-GMM: CTC guided modality matching for fast and accurate streaming speech translationcited by 1
  3. 2024Parameter-efficient Adaptation of Multilingual Multimodal Models for Low-resource ASRcited by 0