# PSST! Prosodic Speech Segmentation with Transformers

arXiv: https://arxiv.org/abs/2302.01984 · HTML: https://mlsanity.com/paper/2302.01984/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 66 references that are in the map.
- First posted on arXiv 3 Feb 2023.

## Builds on (most cited first, 6 of 6)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020)
- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [Phone-to-audio alignment without text: A Semi-supervised Approach](https://mlsanity.com/paper/2110.03876/) (2021)
- [Prosodic Event Recognition using Convolutional Neural Networks with Context Information](https://mlsanity.com/paper/1706.00741/) (2017)

## Built on by (most cited first, 5 of 5)

- [ProsodyFM: Unsupervised Phrasing and Intonation Control for Intelligible Speech Synthesis](https://mlsanity.com/paper/2412.11795/) (2024)
- [CoVoSwitch: Machine Translation of Synthetic Code-Switched Text Based on Intonation Units](https://mlsanity.com/paper/2407.14295/) (2024)
- [Improving French Synthetic Speech Quality via SSML Prosody Control](https://mlsanity.com/paper/2508.17494/) (2025)
- [Word-wise intonation model for cross-language TTS systems](https://mlsanity.com/paper/2409.20374/) (2024)
- [Transformer-based segmentation of prosodic boundaries in Brazilian Portuguese](https://mlsanity.com/paper/2607.07408/) (2026)

## Abstract (arXiv)

Self-attention mechanisms have enabled transformers to achieve superhuman-level performance on many speech-to-text (STT) tasks, yet the challenge of automatic prosodic segmentation has remained unsolved. In this paper we finetune Whisper, a pretrained STT model, to annotate intonation unit (IU) boundaries by repurposing low-frequency tokens. Our approach achieves an accuracy of 95.8%, outperforming previous methods without the need for large-scale labeled data or enterprise grade compute resources. We also diminish input signals by applying a series of filters, finding that low pass filters at a 3.2 kHz level improve segmentation performance in out of sample and out of distribution contexts. We release our model as both a transcription tool and a baseline for further improvements in prosodic segmentation.