mlsanityData to 7 Oct 2026

Which Speech Representation Better Matches Text-Native Reasoning? A Study of Speech-Text Alignment on Frame Rate and Representation

arXiv 2606.12199 ↗First posted 10 Jun 2026eess.AS cs.CL cs.SDcited by 0

On the map: LLM reasoning → Audio-language model reasoning

Abstract (arXiv)

Spoken dialogue models typically start from text LLM backbones, yet reasoning often degrades when conditioning on speech instead of text. We attribute part of this modality gap to a temporal-granularity mismatch: speech tokens are temporally redundant and far longer than text under matched semantics, diluting per-token semantic density and weakening text-native reasoning dynamics. We study speech token design as a representation selection problem and sweep frame rates under a frozen LLM backbone with a fixed information rate. To make low frame rates feasible, we introduce factorized FSQ and a lightweight non-autoregressive audio LM head, scaling capacity to nearly 300,bits/frame without sacrificing efficient prediction. With the bottleneck removed, we sweep frame rates (50rightarrow2.08,Hz) and alignment depth, and observe a consistent best regime for speech QA at 4.17,Hz with intermediate-layer representation alignment.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning. Builds on 42 of its 46 references that are in the map. First posted on arXiv 10 Jun 2026.

Builds on

42 of 46 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Qwen2.5-Omni Technical Report26 Mar
  2. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming29 Aug
  3. 2023AudioChatLlama: Towards General-Purpose Speech Abilities for LLMs12 Nov

Other references · 39

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  7. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  8. 2022Autoregressive Image Generation using Residual Quantizationcited by 706
  9. 2022High Fidelity Neural Audio Compressioncited by 673
  10. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  11. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  12. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
27 more
  1. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  2. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 425
  3. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  4. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 381
  5. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  6. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  7. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 323
  8. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 222
  9. 2024GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbotcited by 218
  10. 2025Kimi-Audio Technical Reportcited by 210
  11. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 169
  12. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 167
  13. 2024Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLMcited by 135
  14. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 123
  15. 2024WavLLM: Towards Robust and Adaptive Speech Large Language Modelcited by 107
  16. 2023Textually Pretrained Speech Language Modelscited by 99
  17. 2023Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLMcited by 92
  18. 2023SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Modelscited by 85
  19. 2025Step-Audio: Unified Understanding and Generation in Intelligent Speech Interactioncited by 83
  20. 2023BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writingcited by 75
  21. 2023Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation taskscited by 71
  22. 2024SLAM-Omni: Timbre-Controllable Voice Interaction System with Single-Stage Trainingcited by 58
  23. 2022SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Datacited by 49
  24. 2023LibriSpeech-PC: Benchmark for Evaluation of Punctuation and Capitalization Capabilities of end-to-end ASR Modelscited by 46
  25. 2024Scaling Speech-Text Pre-training with Synthetic Interleaved Datacited by 45
  26. 2024Distilling an End-to-End Voice Assistant Without Instruction Training Datacited by 35
  27. 2024DM-Codec: Distilling Multimodal Representations for Speech Tokenizationcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Understanding the Modality Gap: An Empirical Study on the Speech-Text Alignment Mechanism of Large Speech Language Models
  2. 2026TADA: A Generative Framework for Speech Modeling via Text-Acoustic Dual Alignment
  3. 2026Closing the Modality Reasoning Gap for Speech Large Language Models
  4. 2025ALAS: An Automatic Latent Alignment Score for Audio Language Models
  5. 2026CORD: Bridging the Audio-Text Reasoning Gap via Weighted On-policy Cross-modal Distillation
  6. 2026Beyond Fixed Frames: Dynamic Character-Aligned Speech Tokenization
  7. 2025Soundwave: Less is More for Speech-Text Alignment in LLMs
  8. 2026Gradient-Based Speech-to-Text Alignment for Any ASR Model: From CTC to Speech LLMs