# PoeticTTS -- Controllable Poetry Reading for Literary Studies

arXiv: https://arxiv.org/abs/2207.05549 · HTML: https://mlsanity.com/paper/2207.05549/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 38 references that are in the map.
- First posted on arXiv 11 Jul 2022.

## Builds on (most cited first, 22 of 22)

- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [MLS: A Large-Scale Multilingual Dataset for Speech Research](https://mlsanity.com/paper/2012.03411/) (2020)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis](https://mlsanity.com/paper/1803.09017/) (2018)
- [Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron](https://mlsanity.com/paper/1803.09047/) (2018)
- [FastPitch: Parallel Text-to-speech with Pitch Prediction](https://mlsanity.com/paper/2006.06873/) (2020)
- [Learning latent representations for style control and transfer in end-to-end speech synthesis](https://mlsanity.com/paper/1812.04342/) (2018)
- [Hi-Fi Multi-Speaker English TTS Dataset](https://mlsanity.com/paper/2104.01497/) (2021)
- [CSS10: A Collection of Single Speaker Speech Datasets for 10 Languages](https://mlsanity.com/paper/1903.11269/) (2019)
- [Fine-grained robust prosody transfer for single-speaker neural text-to-speech](https://mlsanity.com/paper/1907.02479/) (2019)
- [Language-Agnostic Meta-Learning for Low-Resource Text-to-Speech with Articulatory Features](https://mlsanity.com/paper/2203.03191/) (2022)
- [HUI-Audio-Corpus-German: A high quality TTS dataset](https://mlsanity.com/paper/2106.06309/) (2021)
- [Exact Prosody Cloning in Zero-Shot Multispeaker Text-to-Speech](https://mlsanity.com/paper/2206.12229/) (2022)
- [Ctrl-P: Temporal Control of Prosodic Variation for Speech Synthesis](https://mlsanity.com/paper/2106.08352/) (2021)
- [Improving Performance of Seen and Unseen Speech Style Transfer in End-to-end Neural TTS](https://mlsanity.com/paper/2106.10003/) (2021)
- [ADEPT: A Dataset for Evaluating Prosody Transfer](https://mlsanity.com/paper/2106.08321/) (2021)

## Built on by (most cited first, 5 of 5)

- [Low-Resource Multilingual and Zero-Shot Multispeaker TTS](https://mlsanity.com/paper/2210.12223/) (2022)
- [The IMS Toucan System for the Blizzard Challenge 2023](https://mlsanity.com/paper/2310.17499/) (2023)
- [Investigating Stochastic Methods for Prosody Modeling in Speech Synthesis](https://mlsanity.com/paper/2507.00227/) (2025)
- [HaikuS2S: A Cascaded System For Responding In Verse](https://mlsanity.com/paper/2609.23951/) (2026)
- [Not Quite My Tempo: Voice Activity-aware Speech Synthesis for Lip-Synchronous Dubbing](https://mlsanity.com/paper/2609.26486/) (2026)

## Abstract (arXiv)

Speech synthesis for poetry is challenging due to specific intonation patterns inherent to poetic speech. In this work, we propose an approach to synthesise poems with almost human like naturalness in order to enable literary scholars to systematically examine hypotheses on the interplay between text, spoken realisation, and the listener's perception of poems. To meet these special requirements for literary studies, we resynthesise poems by cloning prosodic values from a human reference recitation, and afterwards make use of fine-grained prosody control to manipulate the synthetic speech in a human-in-the-loop setting to alter the recitation w.r.t. specific phenomena. We find that finetuning our TTS model on poetry captures poetic intonation patterns to a large extent which is beneficial for prosody cloning and manipulation and verify the success of our approach both in an objective evaluation as well as in human studies.