mlsanityData to 7 Oct 2026

Closing the Modality Reasoning Gap for Speech Large Language Models

arXiv 2601.05543 ↗First posted 9 Jan 2026cs.CL cs.SD eess.AScited by 9

On the map: LLM reasoning → Audio-language model reasoning

Abstract (arXiv)

Although Speech Large Language Models have achieved notable progress, a substantial modality reasoning gap remains: their reasoning performance on speech inputs is markedly weaker than on text. This gap could be associated with representational drift across Transformer layers and behavior deviations in long-chain reasoning. To address this issue, we introduce TARS, a reinforcement-learning framework that aligns text-conditioned and speech-conditioned trajectories through an asymmetric reward design. The framework employs two dense and complementary signals: representation alignment, which measures layer-wise hidden-state similarity between speech- and text-conditioned trajectories, and behavior alignment, which evaluates semantic consistency between generated outputs and reference text completions. Experiments on challenging reasoning benchmarks, including MMSU and OBQA, show that our approach significantly narrows the modality reasoning gap and achieves state-of-the-art performance among 7B-scale Speech LLMs.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning. Builds on 21 of its 45 references that are in the map. First posted on arXiv 9 Jan 2026.

Builds on

21 of 45 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Qwen2.5-Omni Technical Report26 Mar

Other references · 20

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  3. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  4. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  5. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  6. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  7. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  8. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  9. 2025Qwen3-Omni Technical Reportcited by 451
  10. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 282
  11. 2025Kimi-Audio Technical Reportcited by 210
  12. 2023Listen, Think, and Understandcited by 206
8 more
  1. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 192
  2. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 47
  3. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 34
  4. 2025Closing the Gap Between Text and Speech Understanding in LLMscited by 14
  5. 2025Cross-Modal Knowledge Distillation for Speech Large Language Modelscited by 6
  6. 2025Enhancing Generalization of Speech Large Language Models with Multi-Task Behavior Imitation and Speech-Text Interleavingcited by 3
  7. 2025ALAS: An Automatic Latent Alignment Score for Audio Language Modelscited by 3
  8. 2025Enhancing Speech Large Language Models through Reinforced Behavior Alignmentcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026Anatomy of the Modality Gap: Dissecting the Internal States of End-to-End Speech LLMscited by 6
  2. 2026Generating Synthetic Doctor-Patient Conversations for Long-form Audio Summarizationcited by 4
  3. 2026DiffuSpeech: Silent Thought, Spoken Answer via Unified Speech-Text Diffusioncited by 0
  4. 2026Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoningcited by 0
  5. 2026Entity Binding Failures in Speech LLM Reasoning: Diagnosis and Chain-of-Thought Interventioncited by 0
  6. 2026Is Text All You Need? Text as a Universal Information Bottleneck for Speech LLMscited by 0
  7. 2026Efficient Chain-of-Modality Reasoning via Progressive Compression for Spoken Language Modelscited by 0
  8. 2026The Eloquence submission for Task 2 of the Interspeech 2026 MLC-SLM challengecited by 0
  9. 2026Reducing the Output-Mode Gap in Speech Language Models via Joint-Output On-Policy Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Understanding the Modality Gap: An Empirical Study on the Speech-Text Alignment Mechanism of Large Speech Language Models
  2. 2026CORD: Bridging the Audio-Text Reasoning Gap via Weighted On-policy Cross-modal Distillation
  3. 2026Escape the Language Prior: Mitigating Late-Stage Modality Collapse in Audio Reasoning via Modality-Aware Policy Optimization
  4. 2026Which Speech Representation Better Matches Text-Native Reasoning? A Study of Speech-Text Alignment on Frame Rate and Representation
  5. 2026Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs
  6. 2026X3-OPD: Distilling Reasoning into Large Audio-Language Models via On-Policy Alignment
  7. 2026MAPLE: Modality-Aware Post-training and Learning Ecosystem
  8. 2026Alignment Is All You Need: Instruction-Free Training for General Audio-Language Models