# Period VITS: Variational Inference with Explicit Pitch Modeling for End-to-end Emotional Speech Synthesis

arXiv: https://arxiv.org/abs/2210.15964 · HTML: https://mlsanity.com/paper/2210.15964/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 31 references that are in the map.
- First posted on arXiv 28 Oct 2022.

## Builds on (most cited first, 23 of 23)

- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Least Squares Generative Adversarial Networks](https://mlsanity.com/paper/1611.04076/) (2016)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [Autoencoding beyond pixels using a learned similarity metric](https://mlsanity.com/paper/1512.09300/) (2015)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [End-to-End Adversarial Text-to-Speech](https://mlsanity.com/paper/2006.03575/) (2020)
- [ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit](https://mlsanity.com/paper/1910.10909/) (2019)
- [Parallel Tacotron: Non-Autoregressive and Controllable TTS](https://mlsanity.com/paper/2010.11439/) (2020)
- [Neural source-filter waveform models for statistical parametric speech synthesis](https://mlsanity.com/paper/1904.12088/) (2019)
- [VISinger: Variational Inference with Adversarial Learning for End-to-End Singing Voice Synthesis](https://mlsanity.com/paper/2110.08813/) (2021)
- [Chunked Autoregressive GAN for Conditional Waveform Synthesis](https://mlsanity.com/paper/2110.10139/) (2021)
- [JETS: Jointly Training FastSpeech2 and HiFi-GAN for End to End Text to Speech](https://mlsanity.com/paper/2203.16852/) (2022)
- [Neural source-filter-based waveform model for statistical parametric speech synthesis](https://mlsanity.com/paper/1810.11946/) (2018)
- [Investigation of enhanced Tacotron text-to-speech synthesis systems with self-attention for pitch accent language](https://mlsanity.com/paper/1810.11960/) (2018)
- [PeriodNet: A non-autoregressive waveform generation model with a structure separating periodic and aperiodic components](https://mlsanity.com/paper/2102.07786/) (2021)
- [Unified Source-Filter GAN with Harmonic-plus-Noise Source Excitation Generation](https://mlsanity.com/paper/2205.06053/) (2022)

## Built on by (most cited first, 6 of 6)

- [HiddenSinger: High-Quality Singing Voice Synthesis via Neural Audio Codec and Latent Diffusion Models](https://mlsanity.com/paper/2306.06814/) (2023)
- [PITS: Variational Pitch Inference without Fundamental Frequency for End-to-End Pitch-controllable TTS](https://mlsanity.com/paper/2302.12391/) (2023)
- [Description-based Controllable Text-to-Speech with Cross-Lingual Voice Control](https://mlsanity.com/paper/2409.17452/) (2024)
- [Grapheme-Coherent Phonemic and Prosodic Annotation of Speech by Implicit and Explicit Grapheme Conditioning](https://mlsanity.com/paper/2506.04527/) (2025)
- [Low-Resource Cross-Domain Singing Voice Synthesis via Reduced Self-Supervised Speech Representations](https://mlsanity.com/paper/2402.01520/) (2024)
- [Song Data Cleansing for End-to-End Neural Singer Diarization Using Neural Analysis and Synthesis Framework](https://mlsanity.com/paper/2406.16315/) (2024)

## Abstract (arXiv)

Several fully end-to-end text-to-speech (TTS) models have been proposed that have shown better performance compared to cascade models (i.e., training acoustic and vocoder models separately). However, they often generate unstable pitch contour with audible artifacts when the dataset contains emotional attributes, i.e., large diversity of pronunciation and prosody. To address this problem, we propose Period VITS, a novel end-to-end TTS model that incorporates an explicit periodicity generator. In the proposed method, we introduce a frame pitch predictor that predicts prosodic features, such as pitch and voicing flags, from the input text. From these features, the proposed periodicity generator produces a sample-level sinusoidal source that enables the waveform decoder to accurately reproduce the pitch. Finally, the entire model is jointly optimized in an end-to-end manner with variational inference and adversarial objectives. As a result, the decoder becomes capable of generating more stable, expressive, and natural output waveforms. The experimental results showed that the proposed model significantly outperforms baseline models in terms of naturalness, with improved pitch stability in the generated samples.