mlsanityData to 14 Oct 2026

Cross-Modal Knowledge Distillation for Speech Large Language Models

arXiv 2509.14930 ↗First posted 18 Sep 2025cs.CL cs.AIcited by 6

On the map: Knowledge distillation → Speech and audio distillation

Abstract (arXiv)

In this work, we present the first systematic evaluation of catastrophic forgetting and modality inequivalence in speech large language models, showing that introducing speech capabilities can degrade knowledge and reasoning even when inputs remain textual, and performance further decreases with spoken queries. To address these challenges, we propose a cross-modal knowledge distillation framework that leverages both text-to-text and speech-to-text channels to transfer knowledge from a text-based teacher model to a speech LLM. Extensive experiments on dialogue and audio understanding tasks validate the effectiveness of our approach in preserving textual knowledge, improving cross-modal alignment, and enhancing reasoning in speech-based interactions.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation. Builds on 19 of its 22 references that are in the map. First posted on arXiv 18 Sep 2025.

Builds on

19 of 22 references.

References · 19

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024Qwen2.5 Technical Reportcited by 4,942
  3. 2025Qwen2.5-Omni Technical Reportcited by 736
  4. 2024Qwen2-Audio Technical Reportcited by 560
  5. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
  6. 2019Clotho: An Audio Captioning Datasetcited by 358
  7. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 288
  8. 2025Kimi-Audio Technical Reportcited by 222
  9. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 200
  10. 2024Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLMcited by 141
  11. 2025Step-Audio 2 Technical Reportcited by 105
  12. 2024Seed-ASR: Understanding Diverse Speech and Contexts with LLM-based Speech Recognitioncited by 63
7 more
  1. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 50
  2. 2025FireRedASR: Open-Source Industrial-Grade Mandarin Speech Recognition Models from Encoder-Decoder to LLM Integrationcited by 46
  3. 2025MiDashengLM: Efficient Audio Understanding with General Audio Captionscited by 33
  4. 2025XY-Tokenizer: Mitigating the Semantic-Acoustic Conflict in Low-Bitrate Speech Codecscited by 23
  5. 2025OSUM-EChat: Enhancing End-to-End Empathetic Spoken Chatbot via Understanding-Driven Spoken Dialoguecited by 21
  6. 2025VoxEval: Benchmarking the Knowledge Understanding Capabilities of End-to-End Spoken Language Modelscited by 16
  7. 2025EchoX: Towards Mitigating Acoustic-Semantic Gap via Echo Training for Speech-to-Speech LLMscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2026X-OPD: Cross-Modal On-Policy Distillation for Capability Alignment in Speech LLMscited by 16
  2. 2026Closing the Modality Reasoning Gap for Speech Large Language Modelscited by 9
  3. 2026Raon-Speech Technical Reportcited by 7
  4. 2025Data-Centric Lessons To Improve Speech-Language Pretrainingcited by 3
  5. 2026Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLMcited by 2
  6. 2026LinguDistill: Recovering Linguistic Ability in Vision-Language Models via Selective Cross-Modal Distillationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026From Images to Words: Efficient Cross-Modal Knowledge Distillation to Language Models from Black-box Teachers
  2. 2026Distilling Text LMs into Speech-to-Speech LMs by Matching Internal Text Predictions
  3. 2026X3-OPD: Distilling Reasoning into Large Audio-Language Models via On-Policy Alignment
  4. 2026CORD: Bridging the Audio-Text Reasoning Gap via Weighted On-policy Cross-modal Distillation
  5. 2023Adaptive Knowledge Distillation between Text and Speech Pre-trained Models
  6. 2025DIFFA: Large Language Diffusion Models Can Listen and Understand
  7. 2025Teaching Audio Models to Reason: A Unified Framework for Source- and Layer-wise Distillation
  8. 2023Knowledge Transfer from Pre-trained Language Models to Cif-based Speech Recognizers via Hierarchical Distillation