mlsanityData to 14 Oct 2026

Generative Spoken Language Model based on continuous word-sized audio tokens

arXiv 2310.05224 ↗First posted 8 Oct 2023cs.CL cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

In NLP, text language models based on words or subwords are known to outperform their character-based counterparts. Yet, in the speech community, the standard input of spoken LMs are 20ms or 40ms-long discrete units (shorter than a phoneme). Taking inspiration from word-based LM, we introduce a Generative Spoken Language Model (GSLM) based on word-size continuous-valued audio embeddings that can generate diverse and expressive language output. This is obtained by replacing lookup table for lexical types with a Lexical Embedding function, the cross entropy loss by a contrastive loss, and multinomial sampling by k-NN sampling. The resulting model is the first generative language model based on word-size continuous embeddings. Its performance is on par with discrete unit GSLMs regarding generation quality as measured by automatic metrics and subjective human judgements. Moreover, it is five times more memory efficient thanks to its large 200ms units. In addition, the embeddings before and after the Lexical Embedder are phonetically and semantically interpretable.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 51 of its 82 references that are in the map. First posted on arXiv 8 Oct 2023.

Builds on

51 of 82 references.

References · 51

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  4. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  7. 2017Billion-scale similarity search with GPUscited by 2,497
  8. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  9. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  10. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  11. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  12. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
38 more
  1. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  2. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  3. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  4. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  5. 2013Generating Sequences With Recurrent Neural Networkscited by 554
  6. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 491
  7. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  8. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 354
  9. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 340
  10. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  11. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 309
  12. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  13. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  14. 2022FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesiscited by 136
  15. 2022Generative Spoken Dialogue Language Modelingcited by 126
  16. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  17. 2020The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modelingcited by 101
  18. 2021Scalable Vision Transformers with Hierarchical Poolingcited by 84
  19. 2017Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learningcited by 67
  20. 2018Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speechcited by 52
  21. 2020Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentationcited by 43
  22. 2019The Zero Resource Speech Challenge 2019: TTS without Tcited by 38
  23. 2022Word Discovery in Visually Grounded, Self-Supervised Speech Modelscited by 37
  24. 2022Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challengecited by 34
  25. 2018Truly unsupervised acoustic word embeddings using weak top-down constraints in encoder-decoder modelscited by 29
  26. 2016Discriminative Acoustic Word Embeddings: Recurrent Neural Network-Based Approachescited by 25
  27. 2019VQVAE Unsupervised Unit Discovery and Multi-scale Code2Spec Inverter for Zerospeech Challenge 2019cited by 25
  28. 2022Word Segmentation on Discovered Phone Units with Dynamic Programming and Self-Supervised Scoringcited by 24
  29. 2022Are discrete units necessary for Spoken Language Modeling?cited by 24
  30. 2021Acoustic word embeddings for zero-resource languages using self-supervised contrastive learning and multilingual adaptationcited by 19
  31. 2021ASR4REAL: An extended benchmark for speech modelscited by 14
  32. 2019Unsupervised End-to-End Learning of Discrete Linguistic Units for Voice Conversioncited by 12
  33. 2020A Correspondence Variational Autoencoder for Unsupervised Acoustic Word Embeddingscited by 12
  34. 2020A comparison of self-supervised speech representations as input features for unsupervised acoustic word embeddingscited by 10
  35. 2020Unsupervised Acoustic Unit Representation Learning for Voice Conversion using WaveNet Auto-encoderscited by 8
  36. 2022Are word boundaries useful for unsupervised language learning?cited by 8
  37. 2022Speech Sequence Embeddings using Nearest Neighbors Contrastive Learningcited by 7
  38. 2015Alternative structures for character-level RNNscited by 5

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 7; 2026: 3.

19 papers.

Sort:

By area:

  1. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 126
  2. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  3. 2024Recent Advances in Speech Language Models: A Surveycited by 87
  4. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesiscited by 34
  5. 2024Salmon: A Suite for Acoustic Language Model Evaluationcited by 24
  6. 2024SyllableLM: Learning Coarse Semantic Units for Speech Language Modelscited by 24
  7. 2024Sylber: Syllabic Embedding Representation of Speech from Raw Audiocited by 19
  8. 2024LAST: Language Model Aware Speech Tokenizationcited by 9
  9. 2024Improving Spoken Language Modeling with Phoneme Classification: A Simple Fine-tuning Approachcited by 8
  10. 2025SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervisioncited by 8
  11. 2025PAST: Phonetic-Acoustic Speech Tokenizercited by 6
  12. 2025Spoken Language Modeling with Duration-Penalized Self-Supervised Unitscited by 5
7 more
  1. 2026ZeroSyl: Simple Zero-Resource Syllable Tokenization for Spoken Language Modelingcited by 5
  2. 2024STAB: Speech Tokenizer Assessment Benchmarkcited by 4
  3. 2025Exploring the Effect of Segmentation and Vocabulary Size on Speech Tokenization for Speech Language Modelscited by 4
  4. 2025Data-Centric Lessons To Improve Speech-Language Pretrainingcited by 3
  5. 2026Sylber 2.0: A Universal Syllable Embeddingcited by 2
  6. 2025Discriminating Form and Meaning in Multilingual Models with Minimal-Pair ABX Taskscited by 0
  7. 2026On the Effect of Segmentation Width and Cluster Size on Speech Resynthesis and Continuation in Generative Spoken Language Modelscited by 0