mlsanityData to 14 Oct 2026

Why Do Speech Language Models Fail to Generate Semantically Coherent Outputs? A Modality Evolving Perspective

arXiv 2412.17048 ↗First posted 22 Dec 2024eess.AS cs.CL cs.SDcited by 8

Not on the map. Why?

Abstract (arXiv)

Although text-based large language models exhibit human-level writing ability and remarkable intelligence, speech language models (SLMs) still struggle to generate semantically coherent outputs. There are several potential reasons for this performance degradation: (A) speech tokens mainly provide phonetic information rather than semantic information, (B) the length of speech sequences is much longer than that of text sequences, and (C) paralinguistic information, such as prosody, introduces additional complexity and variability. In this paper, we explore the influence of three key factors separately by transiting the modality from text to speech in an evolving manner. Our findings reveal that the impact of the three factors varies. Factor A has a relatively minor impact, factor B influences syntactical and semantic modeling more obviously, and factor C exerts the most significant impact, particularly in the basic lexical modeling. Based on these findings, we provide insights into the unique challenges of training SLMs and highlight pathways to develop more effective end-to-end SLMs.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 34 of its 40 references that are in the map. First posted on arXiv 22 Dec 2024.

Builds on

34 of 40 references.

References · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  7. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  9. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  10. 2022High Fidelity Neural Audio Compressioncited by 687
  11. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  12. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
22 more
  1. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  2. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  3. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  4. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  5. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  6. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 431
  7. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  8. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 226
  9. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 210
  10. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 126
  11. 2020The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modelingcited by 101
  12. 2023Textually Pretrained Speech Language Modelscited by 101
  13. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 90
  14. 2024Recent Advances in Speech Language Models: A Surveycited by 87
  15. 2023Libriheavy: a 50,000 hours ASR corpus with punctuation casing and contextcited by 85
  16. 2024Self-Supervised Speech Representations are More Phonetic than Semanticcited by 71
  17. 2024Scaling Properties of Speech Language Modelscited by 32
  18. 2024SyllableLM: Learning Coarse Semantic Units for Speech Language Modelscited by 24
  19. 2024Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedbackcited by 23
  20. 2024Sylber: Syllabic Embedding Representation of Speech from Raw Audiocited by 19
  21. 2024IntrinsicVoice: Empowering LLMs with Intrinsic Real-time Voice Interaction Abilitiescited by 17
  22. 2023Acoustic BPE for Speech Generation with Discrete Tokenscited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 5.

8 papers.

  1. 2025Baichuan-Audio: A Unified Framework for End-to-End Speech Interactioncited by 68
  2. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  3. 2025Unlocking Temporal Flexibility: Neural Speech Codec with Variable Frame Ratecited by 4
  4. 2026Quantifying the Generation Modality Gap in Speech-Text Language Modelscited by 1
  5. 2026Self-Guidance: Enhancing Neural Codecs via Decoder Manifold Alignmentcited by 0
  6. 2026Do Spoken Language Models Hear Speech as They Read Text? Bridging Structural Gaps Between Speech and Textcited by 0
  7. 2026StreamAlign: Streaming Text-Aligned Speech Tokenizationcited by 0
  8. 2026LACE: Layer-Wise Compression for Dynamic Frame Rate Codecscited by 0