# fairseq S^2: A Scalable and Integrable Speech Synthesis Toolkit

arXiv: https://arxiv.org/abs/2109.06912 · HTML: https://mlsanity.com/paper/2109.06912/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 67 references that are in the map.
- First posted on arXiv 14 Sep 2021.

## Builds on (most cited first, 30 of 43)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations](https://mlsanity.com/paper/1910.05453/) (2019)
- [Generative Spoken Language Modeling from Raw Audio](https://mlsanity.com/paper/2102.01192/) (2021)
- [WaveGlow: A Flow-based Generative Network for Speech Synthesis](https://mlsanity.com/paper/1811.00002/) (2018)
- [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://mlsanity.com/paper/1806.04558/) (2018)
- [Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis](https://mlsanity.com/paper/1803.09017/) (2018)
- [Speech Resynthesis from Discrete Disentangled Self-Supervised Representations](https://mlsanity.com/paper/2104.00355/) (2021)
- [Real Time Speech Enhancement in the Waveform Domain](https://mlsanity.com/paper/2006.12847/) (2020)
- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [Facebook FAIR's WMT19 News Translation Task Submission](https://mlsanity.com/paper/1907.06616/) (2019)
- [fairseq S2T: Fast Speech-to-Text Modeling with fairseq](https://mlsanity.com/paper/2010.05171/) (2020)
- [Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron](https://mlsanity.com/paper/1803.09047/) (2018)
- [Sequence-to-Sequence Models Can Directly Translate Foreign Speech](https://mlsanity.com/paper/1703.08581/) (2017)
- [Direct speech-to-speech translation with a sequence-to-sequence model](https://mlsanity.com/paper/1904.06037/) (2019)
- [Hierarchical Generative Modeling for Controllable Speech Synthesis](https://mlsanity.com/paper/1810.07217/) (2018)
- [Deep Voice 2: Multi-Speaker Neural Text-to-Speech](https://mlsanity.com/paper/1705.08947/) (2017)
- [Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddings](https://mlsanity.com/paper/1910.10838/) (2019)
- [Speech Recognition with Augmented Synthesized Speech](https://mlsanity.com/paper/1909.11699/) (2019)
- [ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit](https://mlsanity.com/paper/1910.10909/) (2019)
- [Text-Free Image-to-Speech Synthesis Using Learned Segmental Units](https://mlsanity.com/paper/2012.15454/) (2020)

## Built on by (most cited first, 24 of 24)

- [Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale](https://mlsanity.com/paper/2306.15687/) (2023)
- [Textless Speech-to-Speech Translation on Real Data](https://mlsanity.com/paper/2112.08352/) (2021)
- [Textually Pretrained Speech Language Models](https://mlsanity.com/paper/2305.13009/) (2023)
- [Scaling Properties of Speech Language Models](https://mlsanity.com/paper/2404.00685/) (2024)
- [reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputs](https://mlsanity.com/paper/2503.11751/) (2025)
- [ESPnet2-TTS: Extending the Edge of TTS Research](https://mlsanity.com/paper/2110.07840/) (2021)
- [Closing the Gap Between Text and Speech Understanding in LLMs](https://mlsanity.com/paper/2510.13632/) (2025)
- [BLASER: A Text-Free Speech-to-Speech Translation Evaluation Metric](https://mlsanity.com/paper/2212.08486/) (2022)
- [Speaking Style Conversion in the Waveform Domain Using Discrete Self-Supervised Units](https://mlsanity.com/paper/2212.09730/) (2022)
- [Slamming: Training a Speech Language Model on One GPU in a Day](https://mlsanity.com/paper/2502.15814/) (2025)
- [A Holistic Cascade System, benchmark, and Human Evaluation Protocol for Expressive Speech-to-Speech Translation](https://mlsanity.com/paper/2301.10606/) (2023)
- [BERT, can HE predict contrastive focus? Predicting and controlling prominence in neural TTS using a language model](https://mlsanity.com/paper/2207.01718/) (2022)
- [Back Transcription as a Method for Evaluating Robustness of Natural Language Understanding Models to Speech Recognition Errors](https://mlsanity.com/paper/2310.16609/) (2023)
- [DisfluencySpeech -- Single-Speaker Conversational Speech Dataset with Paralanguage](https://mlsanity.com/paper/2406.08820/) (2024)
- [Fake-Mamba: Real-Time Speech Deepfake Detection Using Bidirectional Mamba as Self-Attention's Alternative](https://mlsanity.com/paper/2508.09294/) (2025)
- [From Start to Finish: Latency Reduction Strategies for Incremental Speech Synthesis in Simultaneous Speech-to-Speech Translation](https://mlsanity.com/paper/2110.08214/) (2021)
- [Measuring the Impact of Individual Domain Factors in Self-Supervised Pre-Training](https://mlsanity.com/paper/2203.00648/) (2022)
- [SpeeChain: A Speech Toolkit for Large-Scale Machine Speech Chain](https://mlsanity.com/paper/2301.02966/) (2023)
- [Improving Autoregressive NLP Tasks via Modular Linearized Attention](https://mlsanity.com/paper/2304.08453/) (2023)
- [i-Code V2: An Autoregressive Generation Framework over Vision, Language, and Speech Data](https://mlsanity.com/paper/2305.12311/) (2023)
- [A Language Modeling Approach to Diacritic-Free Hebrew TTS](https://mlsanity.com/paper/2407.12206/) (2024)
- [Energy-Based Models For Speech Synthesis](https://mlsanity.com/paper/2310.12765/) (2023)
- [A Unit-based System and Dataset for Expressive Direct Speech-to-Speech Translation](https://mlsanity.com/paper/2502.00374/) (2025)
- [Attentive Multi-Layer Perceptron for Non-autoregressive Generation](https://mlsanity.com/paper/2310.09512/) (2023)

## Abstract (arXiv)

This paper presents fairseq S^2, a fairseq extension for speech synthesis. We implement a number of autoregressive (AR) and non-AR text-to-speech models, and their multi-speaker variants. To enable training speech synthesis models with less curated data, a number of preprocessing tools are built and their importance is shown empirically. To facilitate faster iteration of development and analysis, a suite of automatic metrics is included. Apart from the features added specifically for this extension, fairseq S^2 also benefits from the scalability offered by fairseq and can be easily integrated with other state-of-the-art systems provided in this framework. The code, documentation, and pre-trained models are available at https://github.com/pytorch/fairseq/tree/master/examples/speech_synthesis.