# Location, Location: Enhancing the Evaluation of Text-to-Speech Synthesis Using the Rapid Prosody Transcription Paradigm

arXiv: https://arxiv.org/abs/2107.02527 · HTML: https://mlsanity.com/paper/2107.02527/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 36 references that are in the map.
- First posted on arXiv 6 Jul 2021.

## Builds on (most cited first, 7 of 7)

- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [FastPitch: Parallel Text-to-speech with Pitch Prediction](https://mlsanity.com/paper/2006.06873/) (2020)
- [Robust and fine-grained prosody control of end-to-end speech synthesis](https://mlsanity.com/paper/1811.02122/) (2018)
- [Dynamic Prosody Generation for Speech Synthesis using Linguistics-Driven Acoustic Embedding Selection](https://mlsanity.com/paper/1912.00955/) (2019)
- [Evaluating Long-form Text-to-Speech: Comparing the Ratings of Sentences and Paragraphs](https://mlsanity.com/paper/1909.03965/) (2019)
- [Using generative modelling to produce varied intonation for speech synthesis](https://mlsanity.com/paper/1906.04233/) (2019)
- [Perception of prosodic variation for speech synthesis using an unsupervised discrete representation of F0](https://mlsanity.com/paper/2003.06686/) (2020)

## Built on by (most cited first, 2 of 2)

- [Pairwise Evaluation of Accent Similarity in Speech Synthesis](https://mlsanity.com/paper/2505.14410/) (2025)
- [Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensions](https://mlsanity.com/paper/2608.09930/) (2026)

## Abstract (arXiv)

Text-to-Speech synthesis systems are generally evaluated using Mean Opinion Score (MOS) tests, where listeners score samples of synthetic speech on a Likert scale. A major drawback of MOS tests is that they only offer a general measure of overall quality-i.e., the naturalness of an utterance-and so cannot tell us where exactly synthesis errors occur. This can make evaluation of the appropriateness of prosodic variation within utterances inconclusive. To address this, we propose a novel evaluation method based on the Rapid Prosody Transcription paradigm. This allows listeners to mark the locations of errors in an utterance in real-time, providing a probabilistic representation of the perceptual errors that occur in the synthetic signal. We conduct experiments that confirm that the fine-grained evaluation can be mapped to system rankings of standard MOS tests, but the error marking gives a much more comprehensive assessment of synthesized prosody. In particular, for standard audiobook test set samples, we see that error marks consistently cluster around words at major prosodic boundaries indicated by punctuation. However, for question-answer based stimuli, where we control information structure, we see differences emerge in the ability of neural TTS systems to generate context-appropriate prosodic prominence.