mlsanityData to 14 Oct 2026

fairseq S^2: A Scalable and Integrable Speech Synthesis Toolkit

arXiv 2109.06912 ↗First posted 14 Sep 2021eess.AS cs.CL cs.SDcited by 24

Not on the map. Why?

Abstract (arXiv)

This paper presents fairseq S^2, a fairseq extension for speech synthesis. We implement a number of autoregressive (AR) and non-AR text-to-speech models, and their multi-speaker variants. To enable training speech synthesis models with less curated data, a number of preprocessing tools are built and their importance is shown empirically. To facilitate faster iteration of development and analysis, a suite of automatic metrics is included. Apart from the features added specifically for this extension, fairseq S^2 also benefits from the scalability offered by fairseq and can be easily integrated with other state-of-the-art systems provided in this framework. The code, documentation, and pre-trained models are available at https://github.com/pytorch/fairseq/tree/master/examples/speech_synthesis.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Builds on 43 of its 67 references that are in the map. First posted on arXiv 14 Sep 2021.

Builds on

43 of 67 references.

References · 43

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2017Neural Discrete Representation Learningcited by 4,712
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  5. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  6. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  7. 2017Mixed Precision Trainingcited by 1,156
  8. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  9. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  10. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  11. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
  12. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 340
31 more
  1. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  2. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 309
  3. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 282
  4. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 259
  5. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  6. 2020Real Time Speech Enhancement in the Waveform Domaincited by 186
  7. 2021A Survey on Neural Speech Synthesiscited by 184
  8. 2019Facebook FAIR's WMT19 News Translation Task Submissioncited by 179
  9. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 176
  10. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 173
  11. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  12. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  13. 2018Hierarchical Generative Modeling for Controllable Speech Synthesiscited by 93
  14. 2017Deep Voice 2: Multi-Speaker Neural Text-to-Speechcited by 75
  15. 2019Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddingscited by 73
  16. 2019Speech Recognition with Augmented Synthesized Speechcited by 60
  17. 2019ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkitcited by 60
  18. 2020Text-Free Image-to-Speech Synthesis Using Learned Segmental Unitscited by 51
  19. 2015End-to-End Text-Dependent Speaker Verificationcited by 49
  20. 2017Listening while Speaking: Speech Chain by Deep Learningcited by 46
  21. 2019Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speechcited by 44
  22. 2020MultiSpeech: Multi-Speaker Text to Speech with Transformercited by 42
  23. 2020Wave-Tacotron: Spectrogram-free end-to-end text-to-speech synthesiscited by 38
  24. 2018Cycle-consistency training for end-to-end speech recognitioncited by 31
  25. 2018Back-Translation-Style Data Augmentation for End-to-End ASRcited by 28
  26. 2020Unsupervised Cross-Domain Singing Voice Conversioncited by 23
  27. 2018Expressive Speech Synthesis via Modeling Expressions with Variational Autoencodercited by 22
  28. 2020DiscreTalk: Text-to-Speech as a Machine Translation Problemcited by 21
  29. 2020DenoiSpeech: Denoising Text to Speech with Frame-Level Noise Modelingcited by 14
  30. 2019Semi-supervised Sequence-to-sequence ASR using Unpaired Speech and Textcited by 10
  31. 2019VizSeq: A Visual Analysis Toolkit for Text Generation Taskscited by 10

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 4; 2023: 9; 2024: 3; 2025: 5.

24 papers.

Sort:

By area:

  1. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 379
  2. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  3. 2023Textually Pretrained Speech Language Modelscited by 101
  4. 2024Scaling Properties of Speech Language Modelscited by 32
  5. 2025reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputscited by 25
  6. 2021ESPnet2-TTS: Extending the Edge of TTS Researchcited by 22
  7. 2025Closing the Gap Between Text and Speech Understanding in LLMscited by 17
  8. 2022BLASER: A Text-Free Speech-to-Speech Translation Evaluation Metriccited by 16
  9. 2022Speaking Style Conversion in the Waveform Domain Using Discrete Self-Supervised Unitscited by 15
  10. 2025Slamming: Training a Speech Language Model on One GPU in a Daycited by 12
  11. 2023A Holistic Cascade System, benchmark, and Human Evaluation Protocol for Expressive Speech-to-Speech Translationcited by 11
  12. 2022BERT, can HE predict contrastive focus? Predicting and controlling prominence in neural TTS using a language modelcited by 10
12 more
  1. 2023Back Transcription as a Method for Evaluating Robustness of Natural Language Understanding Models to Speech Recognition Errorscited by 6
  2. 2024DisfluencySpeech -- Single-Speaker Conversational Speech Dataset with Paralanguagecited by 6
  3. 2025Fake-Mamba: Real-Time Speech Deepfake Detection Using Bidirectional Mamba as Self-Attention's Alternativecited by 6
  4. 2021From Start to Finish: Latency Reduction Strategies for Incremental Speech Synthesis in Simultaneous Speech-to-Speech Translationcited by 3
  5. 2022Measuring the Impact of Individual Domain Factors in Self-Supervised Pre-Trainingcited by 2
  6. 2023SpeeChain: A Speech Toolkit for Large-Scale Machine Speech Chaincited by 2
  7. 2023Improving Autoregressive NLP Tasks via Modular Linearized Attentioncited by 2
  8. 2023i-Code V2: An Autoregressive Generation Framework over Vision, Language, and Speech Datacited by 2
  9. 2024A Language Modeling Approach to Diacritic-Free Hebrew TTScited by 2
  10. 2023Energy-Based Models For Speech Synthesiscited by 1
  11. 2025A Unit-based System and Dataset for Expressive Direct Speech-to-Speech Translationcited by 1
  12. 2023Attentive Multi-Layer Perceptron for Non-autoregressive Generationcited by 0