mlsanityData to 14 Oct 2026

Improving Spoken Language Modeling with Phoneme Classification: A Simple Fine-tuning Approach

arXiv 2410.00025 ↗First posted 16 Sep 2024cs.CL cs.SD eess.AScited by 8

Not on the map. Why?

Abstract (arXiv)

Recent progress in Spoken Language Modeling has shown that learning language directly from speech is feasible. Generating speech through a pipeline that operates at the text level typically loses nuances, intonations, and non-verbal vocalizations. Modeling directly from speech opens up the path to more natural and expressive systems. On the other hand, speech-only systems require up to three orders of magnitude more data to catch up to their text-based counterparts in terms of their semantic abilities. We show that fine-tuning speech representation models on phoneme classification leads to more context-invariant representations, and language models trained on these units achieve comparable lexical comprehension to ones trained on hundred times more data.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 24 of its 31 references that are in the map. First posted on arXiv 16 Sep 2024.

Builds on

24 of 31 references.

References · 24

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  5. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  6. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  7. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  8. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  9. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  10. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  11. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 126
  12. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
12 more
  1. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  2. 2020The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modelingcited by 101
  3. 2023EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesiscited by 98
  4. 2023Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation taskscited by 72
  5. 2022Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challengecited by 34
  6. 2024Scaling Properties of Speech Language Modelscited by 32
  7. 2023Toward Joint Language Modeling for Speech Units and Textcited by 28
  8. 2023Self-supervised Fine-tuning for Improved Content Representations by Speaker-invariant Clusteringcited by 22
  9. 2023Generative Spoken Language Model based on continuous word-sized audio tokenscited by 19
  10. 2022Augmentation Invariant Discrete Representation for Generative Spoken Language Modelingcited by 17
  11. 2022Evaluating context-invariance in unsupervised speech representationscited by 11
  12. 2023R-Spin: Efficient Speaker and Noise-invariant Representation Learning with Acoustic Piecescited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 3.

8 papers.

  1. 2025SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervisioncited by 8
  2. 2025fastabx: A library for efficient computation of ABX discriminabilitycited by 7
  3. 2025IPA-CHILDES & G2P+: Feature-Rich Resources for Cross-Lingual Phonology and Phonemic Language Modelingcited by 6
  4. 2025Spoken Language Modeling with Duration-Penalized Self-Supervised Unitscited by 5
  5. 2025SpidR-Adapt: A Universal Speech Representation Model for Few-Shot Adaptationcited by 1
  6. 2026Scaling Human and G2P Supervision for Robust Phonetic Transcriptioncited by 1
  7. 2026DiscoPhon: Benchmarking the Unsupervised Discovery of Phoneme Inventories With Discrete Speech Unitscited by 0
  8. 2026P2Flow: Phoneme-aware Progressive Flow Matching for Extreme Speech Super-Resolutioncited by 0