# textless-lib: a Library for Textless Spoken Language Processing

arXiv: https://arxiv.org/abs/2202.07359 · HTML: https://mlsanity.com/paper/2202.07359/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 52 references that are in the map.
- First posted on arXiv 15 Feb 2022.

## Builds on (most cited first, 30 of 39)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019)
- [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training](https://mlsanity.com/paper/2108.06209/) (2021)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [Generative Spoken Language Modeling from Raw Audio](https://mlsanity.com/paper/2102.01192/) (2021)
- [WaveGlow: A Flow-based Generative Network for Speech Synthesis](https://mlsanity.com/paper/1811.00002/) (2018)
- [Speech Resynthesis from Discrete Disentangled Self-Supervised Representations](https://mlsanity.com/paper/2104.00355/) (2021)
- [Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks](https://mlsanity.com/paper/1608.04207/) (2016)
- [SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing](https://mlsanity.com/paper/2110.07205/) (2021)
- [Direct speech-to-speech translation with discrete units](https://mlsanity.com/paper/2107.05604/) (2021)
- [Textless Speech-to-Speech Translation on Real Data](https://mlsanity.com/paper/2112.08352/) (2021)
- [Text-Free Prosody-Aware Generative Spoken Language Modeling](https://mlsanity.com/paper/2109.03264/) (2021)
- [CREPE: A Convolutional Representation for Pitch Estimation](https://mlsanity.com/paper/1802.06182/) (2018)
- [The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling](https://mlsanity.com/paper/2011.11588/) (2020)
- [UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data](https://mlsanity.com/paper/2101.07597/) (2021)

## Built on by (most cited first, 28 of 28)

- [AudioLM: a Language Modeling Approach to Audio Generation](https://mlsanity.com/paper/2209.03143/) (2022)
- [AudioGen: Textually Guided Audio Generation](https://mlsanity.com/paper/2209.15352/) (2022)
- [AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head](https://mlsanity.com/paper/2304.12995/) (2023)
- [On The Landscape of Spoken Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2504.08528/) (2025)
- [Textually Pretrained Speech Language Models](https://mlsanity.com/paper/2305.13009/) (2023)
- [EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis](https://mlsanity.com/paper/2308.05725/) (2023)
- [WavChat: A Survey of Spoken Dialogue Models](https://mlsanity.com/paper/2411.13577/) (2024)
- [A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks](https://mlsanity.com/paper/2408.01319/) (2024)
- [Analysing Discrete Self Supervised Speech Representation for Spoken Language Modeling](https://mlsanity.com/paper/2301.00591/) (2023)
- [Make-A-Voice: Unified Voice Synthesis With Discrete Representation](https://mlsanity.com/paper/2305.19269/) (2023)
- [Are discrete units necessary for Spoken Language Modeling?](https://mlsanity.com/paper/2203.05936/) (2022)
- [Augmentation Invariant Discrete Representation for Generative Spoken Language Modeling](https://mlsanity.com/paper/2209.15483/) (2022)
- [UnitSpeech: Speaker-adaptive Speech Synthesis with Untranscribed Data](https://mlsanity.com/paper/2306.16083/) (2023)
- [SpeechPrompt: Prompting Speech Language Models for Speech Processing Tasks](https://mlsanity.com/paper/2408.13040/) (2024)
- [LAST: Language Model Aware Speech Tokenization](https://mlsanity.com/paper/2409.03701/) (2024)
- [Disentangling Prosody Representations with Unsupervised Speech Reconstruction](https://mlsanity.com/paper/2212.06972/) (2022)
- [Improving Textless Spoken Language Understanding with Discrete Units as Intermediate Target](https://mlsanity.com/paper/2305.18096/) (2023)
- [A Textless Metric for Speech-to-Speech Comparison](https://mlsanity.com/paper/2210.11835/) (2022)
- [A unified one-shot prosody and speaker conversion system with self-supervised discrete speech units](https://mlsanity.com/paper/2211.06535/) (2022)
- [Speech-to-Text Translation with Phoneme-Augmented CoT: Enhancing Cross-Lingual Transfer in Low-Resource Scenarios](https://mlsanity.com/paper/2505.24691/) (2025)
- [Whisper-GPT -- Continuous Discrete Hybrid Representation Language Models For Speech And Music](https://mlsanity.com/paper/2412.11449/) (2024)
- [Direct Punjabi to English speech translation using discrete units](https://mlsanity.com/paper/2402.15967/) (2024)
- [Unsupervised Speech Segmentation: A General Approach Using Speech Language Models](https://mlsanity.com/paper/2501.03711/) (2025)
- [MIXAR: Scaling Autoregressive Pixel-based Language Models to Multiple Languages and Scripts](https://mlsanity.com/paper/2604.11575/) (2026)
- [Speech Corpora Divergence Based Unsupervised Data Selection for ASR](https://mlsanity.com/paper/2302.13222/) (2023)
- [Learning Semantic Information from Raw Audio Signal Using Both Contextual and Phonetic Representations](https://mlsanity.com/paper/2402.01298/) (2024)
- [MahaTTS: A Unified Framework for Multilingual Text-to-Speech Synthesis](https://mlsanity.com/paper/2508.14049/) (2025)
- [GmSLM : Generative Marmoset Spoken Language Modeling](https://mlsanity.com/paper/2509.09198/) (2025)

## Abstract (arXiv)

Textless spoken language processing research aims to extend the applicability of standard NLP toolset onto spoken language and languages with few or no textual resources. In this paper, we introduce textless-lib, a PyTorch-based library aimed to facilitate research in this research area. We describe the building blocks that the library provides and demonstrate its usability by discuss three different use-case examples: (i) speaker probing, (ii) speech resynthesis and compression, and (iii) speech continuation. We believe that textless-lib substantially simplifies research the textless setting and will be handful not only for speech researchers but also for the NLP community at large. The code, documentation, and pre-trained models are available at https://github.com/facebookresearch/textlesslib/ .