Energy-Based Models For Speech Synthesis
On the map: Diffusion models → Audio, speech and music
Abstract (arXiv)
Recently there has been a lot of interest in non-autoregressive (non-AR) models for speech synthesis, such as FastSpeech 2 and diffusion models. Unlike AR models, these models do not have autoregressive dependencies among outputs which makes inference efficient. This paper expands the range of available non-AR models with another member called energy-based models (EBMs). The paper describes how noise contrastive estimation, which relies on the comparison between positive and negative samples, can be used to train EBMs. It proposes a number of strategies for generating effective negative samples, including using high-performing AR models. It also describes how sampling from EBMs can be performed using Langevin Markov Chain Monte-Carlo (MCMC). The use of Langevin MCMC enables to draw connections between EBMs and currently popular diffusion models. Experiments on LJSpeech dataset show that the proposed approach offers improvements over Tacotron 2.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 18 of its 30 references that are in the map. First posted on arXiv 19 Oct 2023.
Builds on
18 of 30 references.
Closest ancestors · 1
Same direction, newest first
Other references · 17
- 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
- 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
- 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
- 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
- 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
- 2015Sequence Level Training with Recurrent Neural Networkscited by 568
- 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 308
- 2018Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiencycited by 103
- 2015How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?cited by 76
- 2020Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modelingcited by 65
- 2020Pre-Training Transformers as Energy-Based Cloze Modelscited by 52
- 2019Location-Relative Attention Mechanisms For Robust Long-Form Speech Synthesiscited by 44
5 more
- 2019Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTScited by 33
- 2022Full-band General Audio Synthesis with Score-based Diffusioncited by 32
- 2021fairseq S^2: A Scalable and Integrable Speech Synthesis Toolkitcited by 24
- 2021Residual Energy-Based Models for End-to-End Speech Recognitioncited by 9
- 2022Learn2Sing 2.0: Diffusion and Mutual Information-Based Target Speaker SVS by Learning from Singing Teachercited by 8
Built on by
Citing papers per year: 2025: 1.
1 paper.
- 2025Score-Based Training for Energy-Based TTS Modelscited by 0
- 2025Score-Based Training for Energy-Based TTS Models19 May 2025
Similar, not cited
Close in wording, no citation either way
- 2023Explaining the effects of non-convergent sampling in the training of Energy-Based Models
- 2021PriorGrad: Improving Conditional Denoising Diffusion Models with Data-Dependent Adaptive Prior
- 2023CoMoSpeech: One-Step Speech and Singing Voice Synthesis via Consistency Model
- 2022FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis
- 2023DiffAR: Denoising Diffusion Autoregressive Model for Raw Speech Waveform Generation
- 2025Discrete-Time Diffusion-Like Models for Speech Synthesis
- 2025ECTSpeech: Enhancing Efficient Speech Synthesis via Easy Consistency Tuning
- 2023Schrodinger Bridges Beat Diffusion Models on Text-to-Speech Synthesis