# SHAS: Approaching optimal Segmentation for End-to-End Speech Translation

arXiv: https://arxiv.org/abs/2202.04774 · HTML: https://mlsanity.com/paper/2202.04774/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 29 references that are in the map.
- First posted on arXiv 9 Feb 2022.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [fairseq S2T: Fast Speech-to-Text Modeling with fairseq](https://mlsanity.com/paper/2010.05171/) (2020)
- [Unsupervised Speech Recognition](https://mlsanity.com/paper/2105.11084/) (2021)
- [Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debates](https://mlsanity.com/paper/1911.03167/) (2019)
- [The Multilingual TEDx Corpus for Speech Recognition and Translation](https://mlsanity.com/paper/2102.01757/) (2021)
- [Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?](https://mlsanity.com/paper/2106.01045/) (2021)
- [Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task](https://mlsanity.com/paper/2107.05782/) (2021)
- [Beyond Voice Activity Detection: Hybrid Audio Segmentation for Direct Speech Translation](https://mlsanity.com/paper/2104.11710/) (2021)
- [End-to-End Speech Translation with Pre-trained Models and Adapters: UPC at IWSLT 2021](https://mlsanity.com/paper/2105.04512/) (2021)

## Built on by (most cited first, 26 of 26)

- [How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?](https://mlsanity.com/paper/2412.18495/) (2024)
- [Recent Advances in Direct Speech-to-text Translation](https://mlsanity.com/paper/2306.11646/) (2023)
- [Efficient yet Competitive Speech Translation: FBK@IWSLT2022](https://mlsanity.com/paper/2205.02629/) (2022)
- [MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks](https://mlsanity.com/paper/2507.19634/) (2025)
- [Linguistically Motivated Sign Language Segmentation](https://mlsanity.com/paper/2310.13960/) (2023)
- [MT Metrics Correlate with Human Ratings of Simultaneous Speech Translation](https://mlsanity.com/paper/2211.08633/) (2022)
- [Don't Discard Fixed-Window Audio Segmentation in Speech-to-Text Translation](https://mlsanity.com/paper/2210.13363/) (2022)
- [Direct Speech Translation for Automatic Subtitling](https://mlsanity.com/paper/2209.13192/) (2022)
- [SegAugment: Maximizing the Utility of Speech Translation Data with Segmentation-based Augmentations](https://mlsanity.com/paper/2212.09699/) (2022)
- [Long-Form End-to-End Speech Translation via Latent Alignment Segmentation](https://mlsanity.com/paper/2309.11384/) (2023)
- [Direct Models for Simultaneous Translation and Automatic Subtitling: FBK@IWSLT2023](https://mlsanity.com/paper/2309.15554/) (2023)
- [VivesDebate-Speech: A Corpus of Spoken Argumentation to Leverage Audio Features for Argument Mining](https://mlsanity.com/paper/2302.12584/) (2023)
- [KIT's Offline Speech Translation and Instruction Following Submission for IWSLT 2025](https://mlsanity.com/paper/2505.13036/) (2025)
- [Long-form Simultaneous Speech Translation: Thesis Proposal](https://mlsanity.com/paper/2310.11141/) (2023)
- [How to Evaluate Speech Translation with Source-Aware Neural MT Metrics](https://mlsanity.com/paper/2511.03295/) (2025)
- [End-to-End Single-Channel Speaker-Turn Aware Conversational Speech Translation](https://mlsanity.com/paper/2311.00697/) (2023)
- [FAMA: The First Large-Scale Open-Science Speech Foundation Model for English and Italian](https://mlsanity.com/paper/2505.22759/) (2025)
- [Improved Long-Form Spoken Language Translation with Large Language Models](https://mlsanity.com/paper/2212.09895/) (2022)
- [Speech Translation with Foundation Models and Optimal Transport: UPC at IWSLT23](https://mlsanity.com/paper/2306.01327/) (2023)
- [Lightweight Audio Segmentation for Long-form Speech Translation](https://mlsanity.com/paper/2406.10549/) (2024)
- [Segment Length Matters: A Study of Segment Lengths on Audio Fingerprinting Performance](https://mlsanity.com/paper/2601.17690/) (2026)
- [Streaming Translation and Transcription Through Speech-to-Text Causal Alignment](https://mlsanity.com/paper/2603.11578/) (2026)
- [Multilingual Long-Form Speech Instruction Following: KIT's Submission to IWSLT 2026](https://mlsanity.com/paper/2606.04730/) (2026)
- [DPO-Tuned Large Language Models for Segmentation in Simultaneous Speech Translation](https://mlsanity.com/paper/2510.12195/) (2025)
- [From Slides to Chatbots: Enhancing Large Language Models with University Course Materials](https://mlsanity.com/paper/2510.22272/) (2025)
- [MUSCAT: MUltilingual, SCientific ConversATion Benchmark](https://mlsanity.com/paper/2604.15929/) (2026)

## Abstract (arXiv)

Speech translation models are unable to directly process long audios, like TED talks, which have to be split into shorter segments. Speech translation datasets provide manual segmentations of the audios, which are not available in real-world scenarios, and existing segmentation methods usually significantly reduce translation quality at inference time. To bridge the gap between the manual segmentation of training and the automatic one at inference, we propose Supervised Hybrid Audio Segmentation (SHAS), a method that can effectively learn the optimal segmentation from any manually segmented speech corpus. First, we train a classifier to identify the included frames in a segmentation, using speech representations from a pre-trained wav2vec 2.0. The optimal splitting points are then found by a probabilistic Divide-and-Conquer algorithm that progressively splits at the frame of lowest probability until all segments are below a pre-specified length. Experiments on MuST-C and mTEDx show that the translation of the segments produced by our method approaches the quality of the manual segmentation on 5 language pairs. Namely, SHAS retains 95-98% of the manual segmentation's BLEU score, compared to the 87-93% of the best existing methods. Our method is additionally generalizable to different domains and achieves high zero-shot performance in unseen languages.