# Investigating Stochastic Methods for Prosody Modeling in Speech Synthesis

arXiv: https://arxiv.org/abs/2507.00227 · HTML: https://mlsanity.com/paper/2507.00227/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 29 of its 36 references that are in the map.
- First posted on arXiv 30 Jun 2025.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 29 of 29)

- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [Rectified Flow: A Marginal Preserving Approach to Optimal Transport](https://mlsanity.com/paper/2209.14577/) (2022)
- [NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers](https://mlsanity.com/paper/2304.09116/) (2023)
- [NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Quality](https://mlsanity.com/paper/2205.04421/) (2022)
- [FastPitch: Parallel Text-to-speech with Pitch Prediction](https://mlsanity.com/paper/2006.06873/) (2020)
- [Matcha-TTS: A fast TTS architecture with conditional flow matching](https://mlsanity.com/paper/2309.03199/) (2023)
- [Better speech synthesis through scaling](https://mlsanity.com/paper/2305.07243/) (2023)
- [ASVspoof 5: Design, Collection and Validation of Resources for Spoofing, Deepfake, and Adversarial Attack Detection Using Crowdsourced Speech](https://mlsanity.com/paper/2502.08857/) (2025)
- [A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speech](https://mlsanity.com/paper/2302.04215/) (2023)
- [Language-Agnostic Meta-Learning for Low-Resource Text-to-Speech with Articulatory Features](https://mlsanity.com/paper/2203.03191/) (2022)
- [DelightfulTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2021](https://mlsanity.com/paper/2110.12612/) (2021)
- [DelightfulTTS 2: End-to-End Speech Synthesis with Adversarial Vector-Quantized Auto-Encoders](https://mlsanity.com/paper/2207.04646/) (2022)
- [Exact Prosody Cloning in Zero-Shot Multispeaker Text-to-Speech](https://mlsanity.com/paper/2206.12229/) (2022)
- [Meta Learning Text-to-Speech Synthesis in over 7000 Languages](https://mlsanity.com/paper/2406.06403/) (2024)
- [The IMS Toucan System for the Blizzard Challenge 2023](https://mlsanity.com/paper/2310.17499/) (2023)
- [PoeticTTS -- Controllable Poetry Reading for Literary Studies](https://mlsanity.com/paper/2207.05549/) (2022)
- [Should you use a probabilistic duration model in TTS? Probably! Especially for spontaneous speech](https://mlsanity.com/paper/2406.05401/) (2024)
- [ADEPT: A Dataset for Evaluating Prosody Transfer](https://mlsanity.com/paper/2106.08321/) (2021)
- [Varianceflow: High-Quality and Controllable Text-to-Speech using Variance Information via Normalizing Flow](https://mlsanity.com/paper/2302.13458/) (2023)

## Built on by (most cited first, 2 of 2)

- [Bridging the Stability-Expressivity Gap: Synthetic Data Scaling and Preference Alignment for Low-Resource Spoken Language Models](https://mlsanity.com/paper/2605.27383/) (2026)
- [Mask, Sample, Revise: A Revisable CTMC Inference Stack for Guided Discrete Flow Matching Text-to-Speech](https://mlsanity.com/paper/2606.13989/) (2026)

## Abstract (arXiv)

While generative methods have progressed rapidly in recent years, generating expressive prosody for an utterance remains a challenging task in text-to-speech synthesis. This is particularly true for systems that model prosody explicitly through parameters such as pitch, energy, and duration, which is commonly done for the sake of interpretability and controllability. In this work, we investigate the effectiveness of stochastic methods for this task, including Normalizing Flows, Conditional Flow Matching, and Rectified Flows. We compare these methods to a traditional deterministic baseline, as well as to real human realizations. Our extensive subjective and objective evaluations demonstrate that stochastic methods produce natural prosody on par with human speakers by capturing the variability inherent in human speech. Further, they open up additional controllability options by allowing the sampling temperature to be tuned.