AdaTranS: Adapting with Boundary-based Shrinking for End-to-End Speech Translation
Not on the map. Why?
Abstract (arXiv)
To alleviate the data scarcity problem in End-to-end speech translation (ST), pre-training on data for speech recognition and machine translation is considered as an important technique. However, the modality gap between speech and text prevents the ST model from efficiently inheriting knowledge from the pre-trained models. In this work, we propose AdaTranS for end-to-end ST. It adapts the speech features with a new shrinking mechanism to mitigate the length mismatch between speech and text features by predicting word boundaries. Experiments on the MUST-C dataset demonstrate that AdaTranS achieves better performance than the other shrinking-based methods, with higher inference speed and lower memory usage. Further experiments also show that AdaTranS can be equipped with additional alignment losses to further improve performance.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 27 of its 39 references that are in the map. First posted on arXiv 17 Dec 2022.
Builds on
27 of 39 references.
References · 27
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
- 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
- 2015Attention-Based Models for Speech Recognitioncited by 400
- 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
- 2019CIF: Continuous Integrate-and-Fire for End-to-End Speech Recognitioncited by 76
- 2018Pre-training on high-resource speech recognition improves low-resource speech-to-text translationcited by 70
- 2018End-to-End Automatic Speech Translation of Audiobookscited by 65
- 2022Unified Speech-Text Pre-training for Speech Translation and Recognitioncited by 64
- 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
- 2020Curriculum Pre-training for End-to-End Speech Translationcited by 57
- 2022STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translationcited by 57
15 more
- 2020Bridging the Modality Gap for Speech-to-Text Translationcited by 55
- 2021Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Taskcited by 54
- 2020A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Taskscited by 51
- 2021Learning Shared Semantic Space for Speech-to-Text Translationcited by 49
- 2021Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoderscited by 48
- 2021End-to-end Speech Translation via Cross-modal Progressive Trainingcited by 47
- 2021CTC-based Compression for Direct Speech Translationcited by 45
- 2022SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-trainingcited by 40
- 2021RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformercited by 39
- 2021Learning When to Translate for Streaming Speechcited by 30
- 2020Consecutive Decoding for Speech-to-text Translationcited by 28
- 2020Conv-Transformer Transducer: Low Latency, Low Frame Rate, Streamable End-to-End Speech Recognitioncited by 23
- 2019Exploring Phoneme-Level Speech Representations for End-to-End Speech Translationcited by 15
- 2022Exploring Continuous Integrate-and-Fire for Adaptive Simultaneous Speech Translationcited by 14
- 2020Phone Features Improve Speech Translationcited by 11
Built on by
Citing papers per year: 2023: 2; 2024: 1.
3 papers.