# Controllable Neural Prosody Synthesis

arXiv: https://arxiv.org/abs/2008.03388 · HTML: https://mlsanity.com/paper/2008.03388/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Aug 2020.

## Built on by (most cited first, 10 of 10)

- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [MIDI-DDSP: Detailed Control of Musical Performance via Hierarchical Modeling](https://mlsanity.com/paper/2112.09312/) (2021)
- [The Ethical Implications of Generative Audio Models: A Systematic Literature Review](https://mlsanity.com/paper/2307.05527/) (2023)
- [Context-Aware Prosody Correction for Text-Based Speech Editing](https://mlsanity.com/paper/2102.08328/) (2021)
- [Ctrl-P: Temporal Control of Prosodic Variation for Speech Synthesis](https://mlsanity.com/paper/2106.08352/) (2021)
- [Review of end-to-end speech synthesis technology based on deep learning](https://mlsanity.com/paper/2104.09995/) (2021)
- [A Discourse-level Multi-scale Prosodic Model for Fine-grained Emotion Analysis](https://mlsanity.com/paper/2309.11849/) (2023)
- [Hierarchical Generative Modeling of Melodic Vocal Contours in Hindustani Classical Music](https://mlsanity.com/paper/2408.12658/) (2024)
- [Pseudo-Cepstrum: Pitch Modification for Mel-Based Neural Vocoders](https://mlsanity.com/paper/2512.16519/) (2025)
- [LuxEmo: Expressive Text-to-Speech Corpus for Luxembourgish](https://mlsanity.com/paper/2606.31947/) (2026)

## Abstract (arXiv)

Speech synthesis has recently seen significant improvements in fidelity, driven by the advent of neural vocoders and neural prosody generators. However, these systems lack intuitive user controls over prosody, making them unable to rectify prosody errors (e.g., misplaced emphases and contextually inappropriate emotions) or generate prosodies with diverse speaker excitement levels and emotions. We address these limitations with a user-controllable, context-aware neural prosody generator. Given a real or synthesized speech recording, our model allows a user to input prosody constraints for certain time frames and generates the remaining time frames from input text and contextual prosody. We also propose a pitch-shifting neural vocoder to modify input speech to match the synthesized prosody. Through objective and subjective evaluations we show that we can successfully incorporate user control into our prosody generation model without sacrificing the overall naturalness of the synthesized speech.