mlsanityData to 14 Oct 2026

EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis

arXiv 2308.05725 ↗First posted 10 Aug 2023cs.CL cs.LG cs.SDcited by 98

Not on the map. Why?

Abstract (arXiv)

Recent work has shown that it is possible to resynthesize high-quality speech based, not on text, but on low bitrate discrete units that have been learned in a self-supervised fashion and can therefore capture expressive aspects of speech that are hard to transcribe (prosody, voice styles, non-verbal vocalization). The adoption of these methods is still limited by the fact that most speech synthesis datasets are read, severely limiting spontaneity and expressivity. Here, we introduce Expresso, a high-quality expressive speech dataset for textless speech synthesis that includes both read speech and improvised dialogues rendered in 26 spontaneous expressive styles. We illustrate the challenges and potentials of this dataset with an expressive resynthesis benchmark where the task is to encode the input in low-bitrate units and resynthesize it in a target voice while preserving content and style. We evaluate resynthesis quality with automatic metrics for different self-supervised discrete encoders, and explore tradeoffs between quality, bitrate and invariance to speaker and style. All the dataset, evaluation metrics and baseline models are open source

Key facts. Cited by 98 papers in the map (data through 14 Oct 2026). Builds on 22 of its 29 references that are in the map. First posted on arXiv 10 Aug 2023.

Builds on

22 of 29 references.

References · 22

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  3. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  4. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  5. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  6. 2022High Fidelity Neural Audio Compressioncited by 687
  7. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  8. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  9. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  10. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  11. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  12. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
10 more
  1. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  2. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  3. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 169
  4. 2020The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modelingcited by 101
  5. 2021The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usagecited by 101
  6. 2020Self-training and Pre-training are Complementary for Speech Recognitioncited by 99
  7. 2018The Emotional Voices Database: Towards Controlling the Emotion Dimension in Voice Generation Systemscited by 61
  8. 2022textless-lib: a Library for Textless Spoken Language Processingcited by 28
  9. 2022Augmentation Invariant Discrete Representation for Generative Spoken Language Modelingcited by 17
  10. 2022ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancementcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 30; 2025: 37; 2026: 27.

98 papers.

Sort:

By area:

  1. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 150
  2. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 126
  3. 2023Audiobox: Unified Audio Generation with Natural Language Promptscited by 120
  4. 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokenscited by 113
  5. 2024Recent Advances in Speech Language Models: A Surveycited by 87
  6. 2024Proactive Detection of Voice Cloning with Localized Watermarkingcited by 78
  7. 2026MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interactioncited by 78
  8. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 50
  9. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  10. 2024Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversationcited by 41
  11. 2023HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesiscited by 35
  12. 2024Scaling Properties of Speech Language Modelscited by 32
38 more
  1. 2024DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factorscited by 32
  2. 2023Toward Joint Language Modeling for Speech Units and Textcited by 28
  3. 2024Salmon: A Suite for Acoustic Language Model Evaluationcited by 24
  4. 2023EmphAssess : a Prosodic Benchmark on Assessing Emphasis Transfer in Speech-to-Speech Modelscited by 22
  5. 2024Sylber: Syllabic Embedding Representation of Speech from Raw Audiocited by 19
  6. 2024Long-Form Speech Generation with Spoken Language Modelscited by 19
  7. 2025Scaling Rich Style-Prompted Text-to-Speech Datasetscited by 19
  8. 2024Frozen Large Language Models Can Perceive Paralinguistic Aspects of Speechcited by 18
  9. 2024IndicVoices-R: Unlocking a Massive Multilingual Multi-speaker Speech Corpus for Scaling Indian TTScited by 16
  10. 2024SSR: Alignment-Aware Modality Connector for Speech Language Modelscited by 16
  11. 2025FlowDec: A flow-based full-band general audio codec with high perceptual qualitycited by 15
  12. 2025NVSpeech: An Integrated and Scalable Pipeline for Human-Like Speech Modeling with Paralinguistic Vocalizationscited by 15
  13. 2024Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversationcited by 14
  14. 2025NonverbalTTS: A Public English Corpus of Text-Aligned Nonverbal Vocalizations with Emotion Annotations for Text-to-Speechcited by 14
  15. 2025Audio Super-Resolution with Latent Bridge Modelscited by 14
  16. 2024Talk With Human-like Agents: Empathetic Dialogue Through Perceptible Acoustic Reception and Reactioncited by 13
  17. 2025Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learningcited by 13
  18. 2024Making Flow-Matching-Based Zero-Shot Text-to-Speech Laugh as You Likecited by 11
  19. 2025CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speechcited by 11
  20. 2024Bespoke Non-Stationary Solvers for Fast Sampling of Diffusion and Flow Modelscited by 10
  21. 2024RFWave: Multi-band Rectified Flow for Audio Waveform Reconstructioncited by 9
  22. 2024LAST: Language Model Aware Speech Tokenizationcited by 9
  23. 2024DC-Spin: A Speaker-invariant Speech Tokenizer for Spoken Language Modelscited by 9
  24. 2025OmniChat: Enhancing Spoken Dialogue Systems with Scalable Synthetic Data for Diverse Scenarioscited by 9
  25. 2024Improving Spoken Language Modeling with Phoneme Classification: A Simple Fine-tuning Approachcited by 8
  26. 2025DeepDialogue: A Multi-Turn Emotionally-Rich Spoken Dialogue Datasetcited by 8
  27. 2025SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervisioncited by 8
  28. 2024Rasa: Building Expressive Speech Synthesis Systems for Indian Languages in Low-resource Settingscited by 7
  29. 2025WHISTRESS: Enriching Transcriptions with Sentence Stress Detectioncited by 7
  30. 2025StressTest: Can YOUR Speech LM Handle the Stress?cited by 7
  31. 2025Hearing to Translate: The Effectiveness of Speech Modality Integration into LLMscited by 7
  32. 2024SER Evals: In-domain and Out-of-domain Benchmarking for Speech Emotion Recognitioncited by 6
  33. 2024Gull: A Generative Multifunctional Audio Codeccited by 5
  34. 2024DMOSpeech: Direct Metric Optimization via Distilled Diffusion Model in Zero-Shot Speech Synthesiscited by 5
  35. 2025UniWav: Towards Unified Pre-training for Speech Representation Learning and Generationcited by 5
  36. 2024MELD-ST: An Emotion-aware Speech Translation Datasetcited by 4
  37. 2024SF-Speech: Straightened Flow for Zero-Shot Voice Clonecited by 4
  38. 2025FaceSpeak: Expressive and High-Quality Speech Synthesis from Human Portraits of Different Stylescited by 4

50 most cited of 98 All 98 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Seamless: Multilingual Expressive and Streaming Speech Translation
  2. 2022NoreSpeech: Knowledge Distillation based Conditional Diffusion Model for Noise-robust Expressive TTS
  3. 2026Bridging the Stability-Expressivity Gap: Synthetic Data Scaling and Preference Alignment for Low-Resource Spoken Language Models
  4. 2026Poly-InstructTTS: Learning In-the-Wild Expressive Speech Synthesis from Open-Ended Instructions
  5. 2024Towards Expressive Zero-Shot Speech Synthesis with Hierarchical Prosody Modeling
  6. 2026CuteTTS: Efficient and High-Quality Speech Synthesis via Autoregressive Modeling of Continuous Latents
  7. 2025VoiceRestore: Flow-Matching Transformers for Speech Recording Quality Restoration
  8. 2021On the Interplay Between Sparsity, Naturalness, Intelligibility, and Prosody in Speech Synthesis