Data to 14 Oct 2026

StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models

arXiv 2306.07691 ↗First posted 13 Jun 2023eess.AS cs.AI cs.CLcited by 152

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

In this paper, we present StyleTTS 2, a text-to-speech (TTS) model that leverages style diffusion and adversarial training with large speech language models (SLMs) to achieve human-level TTS synthesis. StyleTTS 2 differs from its predecessor by modeling styles as a latent random variable through diffusion models to generate the most suitable style for the text without requiring reference speech, achieving efficient latent diffusion while benefiting from the diverse speech synthesis offered by diffusion models. Furthermore, we employ large pre-trained SLMs, such as WavLM, as discriminators with our novel differentiable duration modeling for end-to-end training, resulting in improved speech naturalness. StyleTTS 2 surpasses human recordings on the single-speaker LJSpeech dataset and matches it on the multispeaker VCTK dataset as judged by native English speakers. Moreover, when trained on the LibriTTS dataset, our model outperforms previous publicly available models for zero-shot speaker adaptation. This work achieves the first human-level TTS on both single and multispeaker datasets, showcasing the potential of style diffusion and adversarial training with large SLMs. The audio demos and source code are available at https://styletts2.github.io/.

Key facts. Cited by 152 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 57 of its 65 references that are in the map. First posted on arXiv 13 Jun 2023.

Builds on

57 of 65 references.

Closest ancestors · 14

Same direction, newest first

2 more

2021Diff-TTS: A Denoising Diffusion Model for Text-to-Speech3 Apr
2020DiffWave: A Versatile Diffusion Model for Audio Synthesis21 Sep

Other references · 43

2017Attention Is All You Needcited by 60,361
2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 2,010
2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
2017Decoupled Weight Decay Regularizationcited by 1,831
2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
2016Least Squares Generative Adversarial Networkscited by 1,001
2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752

31 more

2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
2020WaveGrad: Estimating Gradients for Waveform Generationcited by 622
2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 298
2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 278
2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 251
2021Projected GANs Converge Fastercited by 208
2021A Survey on Neural Speech Synthesiscited by 184
2021A Variational Perspective on Diffusion-Based Generative Models and Score Matchingcited by 183
2022NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Qualitycited by 152
2020Neural Networks Fail to Learn Periodic Functions and How to Fix Itcited by 100
2020End-to-End Adversarial Text-to-Speechcited by 77
2021UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generationcited by 74
2020Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modelingcited by 65
2021WaveGrad 2: Iterative Refinement for Text-to-Speech Synthesiscited by 65
2021StarGANv2-VC: A Diverse, Unsupervised, Non-parallel Framework for Natural-Sounding Voice Conversioncited by 58
2021PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTScited by 52
2022StyleTTS: A Style-Based Generative Model for Natural and Diverse Text-to-Speech Synthesiscited by 33
2022iSTFTNet: Fast and Lightweight Mel-Spectrogram Vocoder Incorporating Inverse Short-Time Fourier Transformcited by 32
2022VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Featurecited by 32
2021Parallel Tacotron 2: A Non-Autoregressive Neural TTS Model with Differentiable Duration Modelingcited by 31
2022JETS: Jointly Training FastSpeech2 and HiFi-GAN for End to End Text to Speechcited by 26
2023Phoneme-Level BERT for Enhanced Prosody of Text-to-Speech with Grapheme Predictionscited by 23
2022WavThruVec: Latent speech representation as intermediate features for neural speech synthesiscited by 17
2022Mixed-Phoneme BERT: Improving BERT with Mixed Phoneme and Sup-Phoneme Representations for Text to Speechcited by 13
2023Zero-shot text-to-speech synthesis conditioned using self-supervised speech representation modelcited by 10
2023Why We Should Report the Details in Subjective Evaluation of TTS More Rigorouslycited by 10
2022StyleTTS-VC: One-Shot Voice Conversion by Knowledge Transfer from Style-Based TTS Modelscited by 9
2023FoundationTTS: Text-to-Speech for ASR Customization with Generative Language Modelcited by 7
2023M2-CTTS: End-to-End Multi-scale Multi-modal Conversational Text-to-Speech Synthesiscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 45; 2025: 53; 2026: 50.

152 papers.

Sort:

By area:

2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 361
2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 226
2024XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Modelcited by 179
2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 122
2024Large Multimodal Agents: A Surveycited by 105
2025IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speechcited by 90
2024Autoregressive Speech Synthesis without Vector Quantizationcited by 65
2023PromptTTS 2: Describing and Generating Voices with Text Promptcited by 48
2025MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesiscited by 41
2023HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesiscited by 35
2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesiscited by 34
2024DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factorscited by 32

38 more

2025Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidancecited by 27
2024FlashSpeech: Efficient Zero-Shot Speech Synthesiscited by 24
2024Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Surveycited by 23
2024SpeechPrune: Context-aware Token Pruning for Speech Information Retrievalcited by 21
2024DFADD: The Diffusion and Flow-Matching Based Audio Deepfake Datasetcited by 20
2024Just ASR + LLM? A Study on Speech Large Language Models' Ability to Identify and Understand Speaker in Spoken Dialoguecited by 16
2024IndicVoices-R: Unlocking a Massive Multilingual Multi-speaker Speech Corpus for Scaling Indian TTScited by 16
2024TTSDS -- Text-to-Speech Distribution Scorecited by 14
2026MOSS-TTS Technical Reportcited by 14
2024SSDM: Scalable Speech Dysfluency Modelingcited by 13
2024I Can Hear You: Selective Robust Training for Deepfake Audio Detectioncited by 13
2024PeriodWave: Multi-Period Flow Matching for High-Fidelity Waveform Generationcited by 12
2023HiFTNet: A Fast High-Quality Neural Vocoder with Harmonic-plus-Noise Filter and Inverse Short Time Fourier Transformcited by 11
2024Low Frame-rate Speech Codec: a Codec Designed for Fast High-quality Speech LLM Training and Inferencecited by 11
2024Deepfake Media Generation and Detection in the Generative AI Era: A Survey and Outlookcited by 10
2025Prosody-Enhanced Acoustic Pre-training and Acoustic-Disentangled Prosody Adapting for Movie Dubbingcited by 10
2026MMaDA-VLA: Large Diffusion Vision-Language-Action Model with Unified Multi-Modal Instruction and Generationcited by 10
2024Bayesian Parameter-Efficient Fine-Tuning for Overcoming Catastrophic Forgettingcited by 9
2024CLAD: Robust Audio Deepfake Detection Against Manipulation Attacks with Contrastive Learningcited by 9
2025OpenS2S: Advancing Fully Open-Source End-to-End Empathetic Large Speech Language Modelcited by 9
2026CoCoEmo: Composable and Controllable Human-Like Emotional TTS via Activation Steeringcited by 9
2025TTSDS2: Resources and Benchmark for Evaluating Human-Quality Text to Speech Systemscited by 8
2025DMOSpeech 2: Reinforcement Learning for Duration Prediction in Metric-Optimized Speech Synthesiscited by 8
2024Style-Talker: Finetuning Audio Language Model and Style-Based Text-to-Speech Model for Fast Spoken Dialogue Generationcited by 7
2024E1 TTS: Simple and Fast Non-Autoregressive TTScited by 7
2024StyleTTS-ZS: Efficient High-Quality Zero-Shot Text-to-Speech Synthesis with Distilled Time-Varying Style Diffusioncited by 7
2024Rethinking MUSHRA: Addressing Modern Challenges in Text-to-Speech Evaluationcited by 7
2024USAT: A Universal Speaker-Adaptive Text-to-Speech Approachcited by 6
2024Prompting Large Language Models with Audio for General-Purpose Speech Summarizationcited by 6
2024Adversarial Training: A Surveycited by 6
2025Selective Attention Merging for low resource tasks: A case study of Child ASRcited by 6
2025SafeSpeech: Robust and Universal Voice Protection Against Malicious Speech Synthesiscited by 6
2025FlowDubber: Movie Dubbing with LLM-based Semantic-aware Learning and Flow Matching based Voice Enhancingcited by 6
2025VoiceBBQ: Investigating Effect of Content and Acoustics in Social Bias of Spoken Language Modelcited by 6
2024DMOSpeech: Direct Metric Optimization via Distilled Diffusion Model in Zero-Shot Speech Synthesiscited by 5
2025TCSinger 2: Customizable Multilingual Zero-shot Singing Voice Synthesiscited by 5
2025Exploiting Leaderboards for Large-Scale Distribution of Malicious Modelscited by 5
2025SpeechWeave: Diverse Multilingual Synthetic Text & Audio Data Generation Pipeline for Training Text to Speech Modelscited by 5

50 most cited of 152 All 152 (JSON)

Similar, not cited

Close in wording, no citation either way