# CTC-based Compression for Direct Speech Translation

arXiv: https://arxiv.org/abs/2102.01578 · HTML: https://mlsanity.com/paper/2102.01578/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 31 references that are in the map.
- First posted on arXiv 2 Feb 2021.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning](https://mlsanity.com/paper/1609.06773/) (2016)
- [Sequence-to-Sequence Models Can Directly Translate Foreign Speech](https://mlsanity.com/paper/1703.08581/) (2017)
- [Direct speech-to-speech translation with a sequence-to-sequence model](https://mlsanity.com/paper/1904.06037/) (2019)
- [Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation](https://mlsanity.com/paper/1612.01744/) (2016)
- [ESPnet-ST: All-in-One Speech Translation Toolkit](https://mlsanity.com/paper/2004.10234/) (2020)
- [End-to-End Automatic Speech Translation of Audiobooks](https://mlsanity.com/paper/1802.04200/) (2018)
- [A Comparative Study on End-to-end Speech to Text Translation](https://mlsanity.com/paper/1911.08870/) (2019)
- [End-to-End Speech-Translation with Knowledge Distillation: FBK@IWSLT2020](https://mlsanity.com/paper/2006.02965/) (2020)
- [Fast and Accurate Recurrent Neural Network Acoustic Models for Speech Recognition](https://mlsanity.com/paper/1507.06947/) (2015)
- [Exploring Phoneme-Level Speech Representations for End-to-End Speech Translation](https://mlsanity.com/paper/1906.01199/) (2019)
- [On Target Segmentation for Direct Speech Translation](https://mlsanity.com/paper/2009.04707/) (2020)
- [Phone Features Improve Speech Translation](https://mlsanity.com/paper/2005.13681/) (2020)
- [Catplayinginthesnow: Impact of Prior Segmentation on a Model of Visually Grounded Speech](https://mlsanity.com/paper/2006.08387/) (2020)

## Built on by (most cited first, 30 of 45)

- [On decoder-only architecture for speech-to-text and large language model integration](https://mlsanity.com/paper/2307.03917/) (2023)
- [On The Landscape of Spoken Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2504.08528/) (2025)
- [AlignAtt: Using Attention-based Audio-Translation Alignments as a Guide for Simultaneous Speech Translation](https://mlsanity.com/paper/2305.11408/) (2023)
- [Attention as a Guide for Simultaneous Speech Translation](https://mlsanity.com/paper/2212.07850/) (2022)
- [Revisiting End-to-End Speech-to-Text Translation From Scratch](https://mlsanity.com/paper/2206.04571/) (2022)
- [CTC Alignments Improve Autoregressive Translation](https://mlsanity.com/paper/2210.05200/) (2022)
- [Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?](https://mlsanity.com/paper/2402.12025/) (2024)
- [Does Simultaneous Speech Translation need Simultaneous Models?](https://mlsanity.com/paper/2204.03783/) (2022)
- [StreamAtt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History Selection](https://mlsanity.com/paper/2406.06097/) (2024)
- [Speechformer: Reducing Information Loss in Direct Speech Translation](https://mlsanity.com/paper/2109.04574/) (2021)
- [M-Adapter: Modality Adaptation for End-to-End Speech-to-Text Translation](https://mlsanity.com/paper/2207.00952/) (2022)
- [Efficient yet Competitive Speech Translation: FBK@IWSLT2022](https://mlsanity.com/paper/2205.02629/) (2022)
- [When Good and Reproducible Results are a Giant with Feet of Clay: The Importance of Software Quality in NLP](https://mlsanity.com/paper/2303.16166/) (2023)
- [Pushing the Limits of Zero-shot End-to-End Speech Translation](https://mlsanity.com/paper/2402.10422/) (2024)
- [Bayes risk CTC: Controllable CTC alignment in Sequence-to-Sequence tasks](https://mlsanity.com/paper/2210.07499/) (2022)
- [ESPnet-ST-v2: Multipurpose Spoken Language Translation Toolkit](https://mlsanity.com/paper/2304.04596/) (2023)
- [Integrating Pre-Trained Speech and Language Models for End-to-End Speech Recognition](https://mlsanity.com/paper/2312.03668/) (2023)
- [A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Speech Translation](https://mlsanity.com/paper/2406.06937/) (2024)
- [Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoring](https://mlsanity.com/paper/2109.04411/) (2021)
- [Fast-MD: Fast Multi-Decoder End-to-End Speech Translation with Non-Autoregressive Hidden Intermediates](https://mlsanity.com/paper/2109.12804/) (2021)
- [How do Multimodal Foundation Models Encode Text and Speech? An Analysis of Cross-Lingual and Cross-Modal Representations](https://mlsanity.com/paper/2411.17666/) (2024)
- [Direct Speech Translation for Automatic Subtitling](https://mlsanity.com/paper/2209.13192/) (2022)
- [Direct Models for Simultaneous Translation and Automatic Subtitling: FBK@IWSLT2023](https://mlsanity.com/paper/2309.15554/) (2023)
- [Zero-shot Speech Translation](https://mlsanity.com/paper/2107.06010/) (2021)
- [Dodging the Data Bottleneck: Automatic Subtitling with Automatically Segmented ST Corpora](https://mlsanity.com/paper/2209.10608/) (2022)
- [Joint Speech Translation and Named Entity Recognition](https://mlsanity.com/paper/2210.11987/) (2022)
- [Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparison](https://mlsanity.com/paper/2501.02370/) (2025)
- [Can We Achieve High-quality Direct Speech-to-Speech Translation without Parallel Speech Data?](https://mlsanity.com/paper/2406.07289/) (2024)
- [How to Connect Speech Foundation Models and Large Language Models? What Matters and What Does Not](https://mlsanity.com/paper/2409.17044/) (2024)
- [AdaTranS: Adapting with Boundary-based Shrinking for End-to-End Speech Translation](https://mlsanity.com/paper/2212.08911/) (2022)

## Abstract (arXiv)

Previous studies demonstrated that a dynamic phone-informed compression of the input audio is beneficial for speech translation (ST). However, they required a dedicated model for phone recognition and did not test this solution for direct ST, in which a single model translates the input audio into the target language without intermediate representations. In this work, we propose the first method able to perform a dynamic compression of the input indirect ST models. In particular, we exploit the Connectionist Temporal Classification (CTC) to compress the input sequence according to its phonetic characteristics. Our experiments demonstrate that our solution brings a 1.3-1.5 BLEU improvement over a strong baseline on two language pairs (English-Italian and English-German), contextually reducing the memory footprint by more than 10%.