mlsanityData to 7 Oct 2026

CTC-based Compression for Direct Speech Translation

arXiv 2102.01578 ↗First posted 2 Feb 2021cs.CLcited by 45

Not on the map. Why?

Abstract (arXiv)

Previous studies demonstrated that a dynamic phone-informed compression of the input audio is beneficial for speech translation (ST). However, they required a dedicated model for phone recognition and did not test this solution for direct ST, in which a single model translates the input audio into the target language without intermediate representations. In this work, we propose the first method able to perform a dynamic compression of the input indirect ST models. In particular, we exploit the Connectionist Temporal Classification (CTC) to compress the input sequence according to its phonetic characteristics. Our experiments demonstrate that our solution brings a 1.3-1.5 BLEU improvement over a strong baseline on two language pairs (English-Italian and English-German), contextually reducing the memory footprint by more than 10%.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Builds on 20 of its 31 references that are in the map. First posted on arXiv 2 Feb 2021.

Builds on

20 of 31 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  6. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  7. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  8. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 254
  9. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  10. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  11. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  12. 2020ESPnet-ST: All-in-One Speech Translation Toolkitcited by 82
8 more
  1. 2018End-to-End Automatic Speech Translation of Audiobookscited by 65
  2. 2019A Comparative Study on End-to-end Speech to Text Translationcited by 40
  3. 2020End-to-End Speech-Translation with Knowledge Distillation: FBK@IWSLT2020cited by 31
  4. 2015Fast and Accurate Recurrent Neural Network Acoustic Models for Speech Recognitioncited by 26
  5. 2019Exploring Phoneme-Level Speech Representations for End-to-End Speech Translationcited by 15
  6. 2020On Target Segmentation for Direct Speech Translationcited by 12
  7. 2020Phone Features Improve Speech Translationcited by 11
  8. 2020Catplayinginthesnow: Impact of Prior Segmentation on a Model of Visually Grounded Speechcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 14; 2023: 9; 2024: 12; 2025: 5; 2026: 1.

45 papers.

Sort:

  1. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 144
  2. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  3. 2023AlignAtt: Using Attention-based Audio-Translation Alignments as a Guide for Simultaneous Speech Translationcited by 35
  4. 2022Attention as a Guide for Simultaneous Speech Translationcited by 32
  5. 2022Revisiting End-to-End Speech-to-Text Translation From Scratchcited by 26
  6. 2022CTC Alignments Improve Autoregressive Translationcited by 25
  7. 2024Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?cited by 23
  8. 2022Does Simultaneous Speech Translation need Simultaneous Models?cited by 22
  9. 2024StreamAtt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History Selectioncited by 19
  10. 2021Speechformer: Reducing Information Loss in Direct Speech Translationcited by 15
  11. 2022M-Adapter: Modality Adaptation for End-to-End Speech-to-Text Translationcited by 15
  12. 2022Efficient yet Competitive Speech Translation: FBK@IWSLT2022cited by 14
33 more
  1. 2023When Good and Reproducible Results are a Giant with Feet of Clay: The Importance of Software Quality in NLPcited by 14
  2. 2024Pushing the Limits of Zero-shot End-to-End Speech Translationcited by 14
  3. 2022Bayes risk CTC: Controllable CTC alignment in Sequence-to-Sequence taskscited by 12
  4. 2023ESPnet-ST-v2: Multipurpose Spoken Language Translation Toolkitcited by 10
  5. 2023Integrating Pre-Trained Speech and Language Models for End-to-End Speech Recognitioncited by 10
  6. 2024A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Speech Translationcited by 10
  7. 2021Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoringcited by 9
  8. 2021Fast-MD: Fast Multi-Decoder End-to-End Speech Translation with Non-Autoregressive Hidden Intermediatescited by 9
  9. 2024How do Multimodal Foundation Models Encode Text and Speech? An Analysis of Cross-Lingual and Cross-Modal Representationscited by 8
  10. 2022Direct Speech Translation for Automatic Subtitlingcited by 7
  11. 2023Direct Models for Simultaneous Translation and Automatic Subtitling: FBK@IWSLT2023cited by 6
  12. 2021Zero-shot Speech Translationcited by 5
  13. 2022Dodging the Data Bottleneck: Automatic Subtitling with Automatically Segmented ST Corporacited by 5
  14. 2022Joint Speech Translation and Named Entity Recognitioncited by 5
  15. 2024Can We Achieve High-quality Direct Speech-to-Speech Translation without Parallel Speech Data?cited by 4
  16. 2024How to Connect Speech Foundation Models and Large Language Models? What Matters and What Does Notcited by 4
  17. 2025Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparisoncited by 4
  18. 2022AdaTranS: Adapting with Boundary-based Shrinking for End-to-End Speech Translationcited by 3
  19. 2023Cross-Modal Multi-Tasking for Speech-to-Text Translation via Hard Parameter Sharingcited by 3
  20. 2022RedApt: An Adaptor for wav2vec 2 Encoding Faster and Smaller Speech Translation without Quality Compromisecited by 2
  21. 2024CJST: CTC Compressor based Joint Speech and Text Training for Decoder-Only ASRcited by 2
  22. 2025Optimal Transport Regularization for Speech Text Alignment in Spoken Language Modelscited by 2
  23. 2025End-to-end Automatic Speech Recognition and Speech Translation: Integration of Speech Foundational Models and LLMscited by 2
  24. 2022Multiformer: A Head-Configurable Transformer-Based Model for Direct Speech Translationcited by 1
  25. 2022Efficient Speech Translation with Pre-trained Modelscited by 1
  26. 2023Efficient CTC Regularization via Coarse Labels for End-to-End Speech Translationcited by 1
  27. 2023Speech Translation with Foundation Models and Optimal Transport: UPC at IWSLT23cited by 1
  28. 2024How do Hyenas deal with Human Speech? Speech Recognition and Translation with ConfHyenacited by 1
  29. 2024Compact Speech Translation Models via Discrete Speech Units Pretrainingcited by 1
  30. 2024SBAAM! Eliminating Transcript Dependency in Automatic Subtitlingcited by 1
  31. 2024CTC-GMM: CTC guided modality matching for fast and accurate streaming speech translationcited by 1
  32. 2025Improving Language and Modality Transfer in Translation by Character-level Modelingcited by 1
  33. 2026Text-Utilization for Encoder-dominated Speech Recognition Modelscited by 0