mlsanityData to 7 Oct 2026

How Generative Spoken Language Modeling Encodes Noisy Speech: Investigation from Phonetics to Syntactics

arXiv 2306.00697 ↗First posted 1 Jun 2023cs.CL cs.AI eess.AScited by 2

Not on the map. Why?

Abstract (arXiv)

We examine the speech modeling potential of generative spoken language modeling (GSLM), which involves using learned symbols derived from data rather than phonemes for speech analysis and synthesis. Since GSLM facilitates textless spoken language processing, exploring its effectiveness is critical for paving the way for novel paradigms in spoken-language processing. This paper presents the findings of GSLM's encoding and decoding effectiveness at the spoken-language and speech levels. Through speech resynthesis experiments, we revealed that resynthesis errors occur at the levels ranging from phonology to syntactics and GSLM frequently resynthesizes natural but content-altered speech.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 21 of its 31 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

21 of 31 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  7. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  8. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  9. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  10. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  11. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  12. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
9 more
  1. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 381
  2. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  3. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  4. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 196
  5. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 185
  6. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
  7. 2022Probing phoneme, language and speaker information in unsupervised speech representationscited by 31
  8. 2021Discretization and Re-synthesis: an alternative method to solve the Cocktail Party Problemcited by 11
  9. 2022Text-to-speech synthesis from dark data with evaluation-in-the-loop data selectioncited by 4

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1.

2 papers.

  1. 2024LAST: Language Model Aware Speech Tokenizationcited by 9
  2. 2023Do learned speech symbols follow Zipf's law?cited by 3