# Lightweight Audio Segmentation for Long-form Speech Translation

arXiv: https://arxiv.org/abs/2406.10549 · HTML: https://mlsanity.com/paper/2406.10549/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 42 references that are in the map.
- First posted on arXiv 15 Jun 2024.

## Builds on (most cited first, 27 of 27)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [Seamless: Multilingual Expressive and Streaming Speech Translation](https://mlsanity.com/paper/2312.05187/) (2023)
- [The Curious Case of Hallucinations in Neural Machine Translation](https://mlsanity.com/paper/2104.06683/) (2021)
- [Sequence-to-Sequence Models Can Directly Translate Foreign Speech](https://mlsanity.com/paper/1703.08581/) (2017)
- [Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation](https://mlsanity.com/paper/1612.01744/) (2016)
- [Intermediate Loss Regularization for CTC-based Speech Recognition](https://mlsanity.com/paper/2102.03216/) (2021)
- [Speech Translation and the End-to-End Promise: Taking Stock of Where We Are](https://mlsanity.com/paper/2004.06358/) (2020)
- [CTC-Segmentation of Large Corpora for German End-to-end Speech Recognition](https://mlsanity.com/paper/2007.09127/) (2020)
- [SHAS: Approaching optimal Segmentation for End-to-End Speech Translation](https://mlsanity.com/paper/2202.04774/) (2022)
- [JParaCrawl: A Large Scale Web-Based English-Japanese Parallel Corpus](https://mlsanity.com/paper/1911.10668/) (2019)
- [End-to-End Automatic Speech Recognition Integrated With CTC-Based Voice Activity Detection](https://mlsanity.com/paper/2002.00551/) (2020)
- [Beyond Voice Activity Detection: Hybrid Audio Segmentation for Direct Speech Translation](https://mlsanity.com/paper/2104.11710/) (2021)
- [Where's the Point? Self-Supervised Multilingual Punctuation-Agnostic Sentence Segmentation](https://mlsanity.com/paper/2305.18893/) (2023)
- [End-to-End Speech Translation with Pre-trained Models and Adapters: UPC at IWSLT 2021](https://mlsanity.com/paper/2105.04512/) (2021)
- [End-to-end Speech-to-Punctuated-Text Recognition](https://mlsanity.com/paper/2207.03169/) (2022)
- [Speech Segmentation Optimization using Segmented Bilingual Speech Corpus for End-to-end Speech Translation](https://mlsanity.com/paper/2203.15479/) (2022)
- [SegAugment: Maximizing the Utility of Speech Translation Data with Segmentation-based Augmentations](https://mlsanity.com/paper/2212.09699/) (2022)
- [Long-Form End-to-End Speech Translation via Latent Alignment Segmentation](https://mlsanity.com/paper/2309.11384/) (2023)
- [Segmenting Subtitles for Correcting ASR Segmentation Errors](https://mlsanity.com/paper/2104.07868/) (2021)
- [E2E Segmentation in a Two-Pass Cascaded Encoder ASR Model](https://mlsanity.com/paper/2211.15432/) (2022)
- [Improved Training for End-to-End Streaming Automatic Speech Recognition Model with Punctuation](https://mlsanity.com/paper/2306.01296/) (2023)
- [Improving Speech Translation Accuracy and Time Efficiency with Fine-tuned wav2vec 2.0-based Speech Segmentation](https://mlsanity.com/paper/2304.12659/) (2023)

## Built on by (most cited first, 1 of 1)

- [Segment Length Matters: A Study of Segment Lengths on Audio Fingerprinting Performance](https://mlsanity.com/paper/2601.17690/) (2026)

## Abstract (arXiv)

Speech segmentation is an essential part of speech translation (ST) systems in real-world scenarios. Since most ST models are designed to process speech segments, long-form audio must be partitioned into shorter segments before translation. Recently, data-driven approaches for the speech segmentation task have been developed. Although the approaches improve overall translation quality, a performance gap exists due to a mismatch between the models and ST systems. In addition, the prior works require large self-supervised speech models, which consume significant computational resources. In this work, we propose a segmentation model that achieves better speech translation quality with a small model size. We propose an ASR-with-punctuation task as an effective pre-training strategy for the segmentation model. We also show that proper integration of the speech segmentation model into the underlying ST system is critical to improve overall translation quality at inference time.