mlsanityData to 14 Oct 2026

CAAD: Contrastive Audio-Aware Distillation for Efficient Speech Language Models

arXiv 2606.23052 ↗First posted 22 Jun 2026eess.AScited by 2

On the map: Knowledge distillation → Speech and audio distillation

Abstract (arXiv)

Speech Language Models achieve reasoning capabilities, but are often hindered by massive parameter counts and a tendency to prioritize linguistic priors over acoustic features. While contrastive decoding enhances grounding by contrasting audio-aware and text-only logits, it increases inference latency. We propose Contrastive Audio-Aware Distillation (CAAD), a framework that internalizes the teacher's contrastive reasoning into the student model's weights. To overcome the high computational training overhead in the dual-path token-by-token contrastive distillation process, we introduce a synchronized teacher-forcing strategy. Anchored by unified Pseudo-Ground Truths, this mechanism enables simultaneous full-sequence generation of the teacher's contrastive distributions, allowing student to distill the audio-aware signal efficiently. Overall, CAAD yields a ~8% relative gain over standard knowledge distillation on Dynamic-SUPERB and successfully reduces linguistic bias in MCR-BENCH.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation. Builds on 32 of its 34 references that are in the map. First posted on arXiv 22 Jun 2026.

Builds on

32 of 34 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Efficient Interleaved Speech Modeling through Knowledge Distillation30 Jun

Other references · 31

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  3. 2025Qwen2.5-Omni Technical Reportcited by 736
  4. 2017VoxCeleb: a large-scale speaker identification datasetcited by 732
  5. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 643
  6. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  7. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  8. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 455
  9. 2025Kimi-Audio Technical Reportcited by 222
  10. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 205
  11. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  12. 2022PromptTTS: Controllable Text-to-Speech with Text Descriptionscited by 110
19 more
  1. 2023Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speechcited by 101
  2. 2024Recent Advances in Speech Language Models: A Surveycited by 87
  3. 2022DailyTalk: Spoken Dialogue Dataset for Conversational Text-to-Speechcited by 69
  4. 2024The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audiocited by 53
  5. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 50
  6. 2024DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Datacited by 45
  7. 2025MLLMs are Deeply Affected by Modality Biascited by 40
  8. 2024DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignmentcited by 33
  9. 2024Understanding Sounds, Missing the Questions: The Challenge of Object Hallucination in Large Audio-Language Modelscited by 31
  10. 2025Reducing Object Hallucination in Large Audio-Language Models via Audio-Aware Decodingcited by 24
  11. 2022Brouhaha: multi-task training for voice activity detection, speech-to-noise ratio, and C50 room acoustics estimationcited by 23
  12. 2024Building a Taiwanese Mandarin Spoken Language Model: A First Attemptcited by 23
  13. 2020AccentDB: A Database of Non-Native English Accents to Assist Neural Speech Recognitioncited by 19
  14. 2025When Audio and Text Disagree: Revealing Text Bias in Large Audio-Language Modelscited by 13
  15. 2023CoLLD: Contrastive Layer-to-layer Distillation for Compressing Multilingual Pre-trained Speech Encoderscited by 9
  16. 2024CKD: Contrastive Knowledge Distillation from A Sample-wise Perspectivecited by 7
  17. 2025Text Takes Over: A Study of Modality Bias in Multimodal Intent Detectioncited by 7
  18. 2026How Contrastive Decoding Enhances Large Audio Language Modelscited by 6
  19. 2026When Audio-LLMs Don't Listen: A Cross-Linguistic Study of Modality Arbitrationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Tri-PvP: Exposing Modality Bias in Omni-Modal Large Language Models through Perceptual-Propositional Evidence Conflictscited by 0
  2. 2026Reward-Tilted On-Policy Distillation for Acoustic Grounding in Audio-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026X3-OPD: Distilling Reasoning into Large Audio-Language Models via On-Policy Alignment
  2. 2025Teaching Audio Models to Reason: A Unified Framework for Source- and Layer-wise Distillation
  3. 2024Distillation Contrastive Decoding: Improving LLMs Reasoning with Contrastive Decoding and Distillation
  4. 2026CORD: Bridging the Audio-Text Reasoning Gap via Weighted On-policy Cross-modal Distillation
  5. 2026ASKD-Whisper: Adaptive Self-knowledge Distillation for Efficient and Low-Latency Automatic Speech Recognition
  6. 2026EchoDistill: Robust Large Audio Language Models via Noisy-to-Clean Self-Distillation
  7. 2026OP-CAD: On-Policy Clean-Audio Distillation for Robust Audio-Visual Reasoning
  8. 2025DIFFA: Large Language Diffusion Models Can Listen and Understand