mlsanityData to 14 Oct 2026

Bridging the Gaps of Both Modality and Language: Synchronous Bilingual CTC for Speech Translation and Speech Recognition

arXiv 2309.12234 ↗First posted 21 Sep 2023cs.CL eess.AScited by 0

Not on the map. Why?

Abstract (arXiv)

In this study, we present synchronous bilingual Connectionist Temporal Classification (CTC), an innovative framework that leverages dual CTC to bridge the gaps of both modality and language in the speech translation (ST) task. Utilizing transcript and translation as concurrent objectives for CTC, our model bridges the gap between audio and text as well as between source and target languages. Building upon the recent advances in CTC application, we develop an enhanced variant, BiL-CTC+, that establishes new state-of-the-art performances on the MuST-C ST benchmarks under resource-constrained scenarios. Intriguingly, our method also yields significant improvements in speech recognition performance, revealing the effect of cross-lingual learning on transcription and demonstrating its broad applicability. The source code is available at https://github.com/xuchennlp/S2T.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 25 references that are in the map. First posted on arXiv 21 Sep 2023.

Builds on

19 of 25 references.

References · 19

  1. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  2. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  3. 2016Sequence-Level Knowledge Distillationcited by 801
  4. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  5. 2021Intermediate Loss Regularization for CTC-based Speech Recognitioncited by 96
  6. 2018Pre-training on high-resource speech recognition improves low-resource speech-to-text translationcited by 70
  7. 2021Relaxing the Conditional Independence Assumption of CTC-based ASR by Conditioning on Intermediate Predictionscited by 61
  8. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
  9. 2022STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translationcited by 57
  10. 2021Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoderscited by 48
  11. 2021End-to-end Speech Translation via Cross-modal Progressive Trainingcited by 47
  12. 2019A Comparative Study on End-to-end Speech to Text Translationcited by 40
7 more
  1. 2023Pre-training for Speech Translation: CTC Meets Optimal Transportcited by 25
  2. 2023CMOT: Cross-modal Mixup via Optimal Transport for Speech Translationcited by 22
  3. 2022M3ST: Mix at Three Levels for Speech Translationcited by 21
  4. 2023Understanding and Bridging the Modality Gap for Speech Translationcited by 15
  5. 2023Recent Advances in Direct Speech-to-text Translationcited by 15
  6. 2023CTC-based Non-autoregressive Speech Translationcited by 9
  7. 2023Improving Speech Translation by Cross-Modal Multi-Grained Contrastive Learningcited by 7

Built on by

Not cited yet.