mlsanityData to 7 Oct 2026

Speech Resynthesis from Discrete Disentangled Self-Supervised Representations

arXiv 2104.00355 ↗First posted 1 Apr 2021cs.SD cs.LG eess.AScited by 243

Not on the map. Why?

Abstract (arXiv)

We propose using self-supervised discrete representations for the task of speech resynthesis. To generate disentangled representation, we separately extract low-bitrate representations for speech content, prosodic information, and speaker identity. This allows to synthesize speech in a controllable manner. We analyze various state-of-the-art, self-supervised representation learning methods and shed light on the advantages of each method while considering reconstruction quality and disentanglement properties. Specifically, we evaluate the F0 reconstruction, speaker identification performance (for both resynthesis and voice conversion), recordings' intelligibility, and overall quality using subjective human evaluation. Lastly, we demonstrate how these representations can be used for an ultra-lightweight speech codec. Using the obtained representations, we can get to a rate of 365 bits per second while providing better speech quality than the baseline methods. Audio samples can be found under the following link: speechbot.github.io/resynthesis.

Key facts. Cited by 243 papers in the map (data through 7 Oct 2026). Builds on 32 of its 52 references that are in the map. First posted on arXiv 1 Apr 2021.

Builds on

32 of 52 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2017Neural Discrete Representation Learningcited by 4,650
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  7. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  8. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  9. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
  10. 2020Jukebox: A Generative Model for Musiccited by 535
  11. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  12. 2015Autoencoding beyond pixels using a learned similarity metriccited by 407
20 more
  1. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  2. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 308
  3. 2018LPCNet: Improving Neural Speech Synthesis Through Linear Predictioncited by 109
  4. 2017Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Datacited by 92
  5. 2020Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challengecited by 55
  6. 2015End-to-End Text-Dependent Speaker Verificationcited by 49
  7. 2019Low Bit-Rate Speech Coding with VQ-VAE and a WaveNet Decodercited by 47
  8. 2020Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentationcited by 43
  9. 2020The Zero Resource Speech Challenge 2020: Discovering discrete subword and word unitscited by 39
  10. 2019The Zero Resource Speech Challenge 2019: TTS without Tcited by 38
  11. 2017Wavenet based low rate speech codingcited by 32
  12. 2020Unsupervised Cross-Domain Singing Voice Conversioncited by 23
  13. 2020Transformer VQ-VAE for Unsupervised Unit Discovery and Speech Synthesis: ZeroSpeech 2020 Challengecited by 21
  14. 2019Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networkscited by 20
  15. 2019A Real-Time Wideband Neural Vocoder at 1.6 kb/s Using LPCNetcited by 16
  16. 2019TTS Skins: Speaker Conversion via ASRcited by 14
  17. 2021High Fidelity Speech Regeneration with Application to Speech Enhancementcited by 10
  18. 2019Improving Opus Low Bit Rate Quality with Neural Speech Synthesiscited by 9
  19. 2016Speex: A Free Codec For Free Speechcited by 6
  20. 2020Improved Prosody from Learned F0 Codebook Representations for VQ-VAE Speech Waveform Reconstructioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 39; 2023: 65; 2024: 65; 2025: 42; 2026: 18.

243 papers.

Sort:

By area:

  1. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  2. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  3. 2022High Fidelity Neural Audio Compressioncited by 673
  4. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  5. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  6. 2022AudioGen: Textually Guided Audio Generationcited by 292
  7. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  8. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 222
  9. 2021A Survey on Neural Speech Synthesiscited by 182
  10. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 169
  11. 2023Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesiscited by 150
  12. 2021Direct speech-to-speech translation with discrete unitscited by 148
38 more
  1. 2022Generative Spoken Dialogue Language Modelingcited by 125
  2. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 123
  3. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  4. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  5. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  6. 2023Textually Pretrained Speech Language Modelscited by 99
  7. 2023EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesiscited by 93
  8. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 88
  9. 2023SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Modelscited by 85
  10. 2024Recent Advances in Speech Language Models: A Surveycited by 85
  11. 2021Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representationscited by 81
  12. 2022ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakerscited by 73
  13. 2022General-purpose, long-context autoregressive modeling with Perceiver ARcited by 60
  14. 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
  15. 2023Generative Pre-training for Speech with Flow Matchingcited by 55
  16. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
  17. 2024Masked Audio Generation using a Single Non-Autoregressive Transformercited by 50
  18. 2024Towards Robust Speech Representation Learning for Thousands of Languagescited by 48
  19. 2024Scaling Transformers for Low-Bitrate High-Quality Speech Codingcited by 48
  20. 2023Analysing Discrete Self Supervised Speech Representation for Spoken Language Modelingcited by 47
  21. 2022Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentationcited by 43
  22. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  23. 2024Zero-shot Voice Conversion with Diffusion Transformerscited by 39
  24. 2023DDDM-VC: Decoupled Denoising Diffusion Models with Disentangled Representation and Prior Mixup for Verified Robust Voice Conversioncited by 38
  25. 2024SpeechGPT-Gen: Scaling Chain-of-Information Speech Generationcited by 36
  26. 2022SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translationscited by 35
  27. 2022TranSpeech: Speech-to-Speech Translation With Bilateral Perturbationcited by 34
  28. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesiscited by 34
  29. 2022An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Eracited by 32
  30. 2024DASB - Discrete Audio and Speech Benchmarkcited by 32
  31. 2023Diff-HierVC: Diffusion-based Hierarchical Voice Conversion with Robust Pitch Generation and Masked Prior for Zero-shot Speaker Adaptationcited by 30
  32. 2023OpenVoice: Versatile Instant Voice Cloningcited by 30
  33. 2021Textless Speech Emotion Conversion using Discrete and Decomposed Representationscited by 29
  34. 2022textless-lib: a Library for Textless Spoken Language Processingcited by 28
  35. 2023Intelligible Lip-to-Speech Synthesis with Speech Unitscited by 28
  36. 2023Make-A-Voice: Unified Voice Synthesis With Discrete Representationcited by 27
  37. 2023RepCodec: A Speech Representation Codec for Speech Tokenizationcited by 27
  38. 2023Toward Joint Language Modeling for Speech Units and Textcited by 27

50 most cited of 243 All 243 (JSON)