# Closing the Modality Reasoning Gap for Speech Large Language Models

arXiv: https://arxiv.org/abs/2601.05543 · HTML: https://mlsanity.com/paper/2601.05543/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning.
- Builds on 21 of its 45 references that are in the map.
- First posted on arXiv 9 Jan 2026.

## Placed in

- LLM reasoning / [Audio-language model reasoning](https://mlsanity.com/lr/audio-language-model-reasoning/)

## Builds on (most cited first, 21 of 21)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [Qwen2.5-Omni Technical Report](https://mlsanity.com/paper/2503.20215/) (2025)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://mlsanity.com/paper/2311.07919/) (2023)
- [SALMONN: Towards Generic Hearing Abilities for Large Language Models](https://mlsanity.com/paper/2310.13289/) (2023)
- [Qwen3-Omni Technical Report](https://mlsanity.com/paper/2509.17765/) (2025)
- [UnifiedQA: Crossing Format Boundaries With a Single QA System](https://mlsanity.com/paper/2005.00700/) (2020)
- [CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models](https://mlsanity.com/paper/2412.10117/) (2024)
- [Kimi-Audio Technical Report](https://mlsanity.com/paper/2504.18425/) (2025)
- [Listen, Think, and Understand](https://mlsanity.com/paper/2305.10790/) (2023)
- [VoiceBench: Benchmarking LLM-Based Voice Assistants](https://mlsanity.com/paper/2410.17196/) (2024)
- [DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignment](https://mlsanity.com/paper/2507.02768/) (2025)
- [Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle](https://mlsanity.com/paper/2509.16679/) (2025)
- [Closing the Gap Between Text and Speech Understanding in LLMs](https://mlsanity.com/paper/2510.13632/) (2025)
- [Cross-Modal Knowledge Distillation for Speech Large Language Models](https://mlsanity.com/paper/2509.14930/) (2025)
- [Enhancing Generalization of Speech Large Language Models with Multi-Task Behavior Imitation and Speech-Text Interleaving](https://mlsanity.com/paper/2505.18644/) (2025)
- [ALAS: An Automatic Latent Alignment Score for Audio Language Models](https://mlsanity.com/paper/2505.19937/) (2025)
- [Enhancing Speech Large Language Models through Reinforced Behavior Alignment](https://mlsanity.com/paper/2509.03526/) (2025)

## Built on by (most cited first, 9 of 9)

- [Geometric Alignment without Functional Equivalence: A Layer-wise Analysis of the Speech-Text Modality Gap](https://mlsanity.com/paper/2603.01502/) (2026)
- [Generating Synthetic Doctor-Patient Conversations for Long-form Audio Summarization](https://mlsanity.com/paper/2604.06138/) (2026)
- [Entity Binding Failures in Speech LLM Reasoning: Diagnosis and Chain-of-Thought Intervention](https://mlsanity.com/paper/2606.04474/) (2026)
- [Is Text All You Need? Text as a Universal Information Bottleneck for Speech LLMs](https://mlsanity.com/paper/2606.09366/) (2026)
- [DiffuSpeech: Silent Thought, Spoken Answer via Unified Speech-Text Diffusion](https://mlsanity.com/paper/2601.22889/) (2026)
- [Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning](https://mlsanity.com/paper/2606.03234/) (2026)
- [Efficient Chain-of-Modality Reasoning via Progressive Compression for Spoken Language Models](https://mlsanity.com/paper/2607.19932/) (2026)
- [The Eloquence submission for Task 2 of the Interspeech 2026 MLC-SLM challenge](https://mlsanity.com/paper/2609.11724/) (2026)
- [Reducing the Output-Mode Gap in Speech Language Models via Joint-Output On-Policy Distillation](https://mlsanity.com/paper/2609.15313/) (2026)

## Abstract (arXiv)

Although Speech Large Language Models have achieved notable progress, a substantial modality reasoning gap remains: their reasoning performance on speech inputs is markedly weaker than on text. This gap could be associated with representational drift across Transformer layers and behavior deviations in long-chain reasoning. To address this issue, we introduce TARS, a reinforcement-learning framework that aligns text-conditioned and speech-conditioned trajectories through an asymmetric reward design. The framework employs two dense and complementary signals: representation alignment, which measures layer-wise hidden-state similarity between speech- and text-conditioned trajectories, and behavior alignment, which evaluates semantic consistency between generated outputs and reference text completions. Experiments on challenging reasoning benchmarks, including MMSU and OBQA, show that our approach significantly narrows the modality reasoning gap and achieves state-of-the-art performance among 7B-scale Speech LLMs.