mlsanityData to 7 Oct 2026

SALTTS: Leveraging Self-Supervised Speech Representations for improved Text-to-Speech Synthesis

arXiv 2308.01018 ↗First posted 2 Aug 2023cs.CL cs.SD eess.AScited by 0

Not on the map. Why?

Abstract (arXiv)

While FastSpeech2 aims to integrate aspects of speech such as pitch, energy, and duration as conditional inputs, it still leaves scope for richer representations. As a part of this work, we leverage representations from various Self-Supervised Learning (SSL) models to enhance the quality of the synthesized speech. In particular, we pass the FastSpeech2 encoder's length-regulated outputs through a series of encoder layers with the objective of reconstructing the SSL representations. In the SALTTS-parallel implementation, the representations from this second encoder are used for an auxiliary reconstruction loss with the SSL features. The SALTTS-cascade implementation, however, passes these representations through the decoder in addition to having the reconstruction loss. The richness of speech characteristics from the SSL features reflects in the output speech quality, with the objective and subjective evaluation measures of the proposed approach outperforming the baseline FastSpeech2.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 20 of its 21 references that are in the map. First posted on arXiv 2 Aug 2023.

Builds on

20 of 21 references.

References · 20

  1. 2022Generative Adversarial Networkscited by 5,603
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  5. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  6. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  7. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  8. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  9. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  10. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  11. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 269
  12. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
8 more
  1. 2022NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Qualitycited by 151
  2. 2021Direct speech-to-speech translation with discrete unitscited by 148
  3. 2022SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilitiescited by 76
  4. 2022VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Featurecited by 32
  5. 2022Towards Building Text-To-Speech Systems for the Next Billion Userscited by 27
  6. 2022JETS: Jointly Training FastSpeech2 and HiFi-GAN for End to End Text to Speechcited by 26
  7. 2022CCC-wav2vec 2.0: Clustering aided Cross Contrastive Self-supervised learning of speech representationscited by 11
  8. 2022data2vec-aqc: Search for the right Teaching Assistant in the Teacher-Student training setupcited by 4

Built on by

Not cited yet.