mlsanityData to 14 Oct 2026

Efficient Speech Translation through Model Compression and Knowledge Distillation

arXiv 2505.20237 ↗First posted 26 May 2025cs.CL cs.SD eess.AScited by 5

On the map: Knowledge distillation → Speech and audio distillation

Abstract (arXiv)

Efficient deployment of large audio-language models for speech translation remains challenging due to their significant computational requirements. In this paper, we address this challenge through our system submissions to the "Model Compression" track at the International Conference on Spoken Language Translation (IWSLT 2025). We experiment with a combination of approaches including iterative layer pruning based on layer importance evaluation, low-rank adaptation with 4-bit quantization (QLoRA), and knowledge distillation. In our experiments, we use Qwen2-Audio-7B-Instruct for speech translation into German and Chinese. Our pruned (student) models achieve up to a 50% reduction in both model parameters and storage footprint, while retaining 97-100% of the translation quality of the in-domain (teacher) models.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation. Builds on 19 of its 29 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

19 of 29 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling1 Nov

Other references · 18

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  4. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  5. 2016Sequence-Level Knowledge Distillationcited by 801
  6. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  7. 2021Datasets: A Community Library for Natural Language Processingcited by 530
  8. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 243
  9. 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scalingcited by 172
  10. 2020On the Effect of Dropping Layers of Pre-trained Transformer Modelscited by 118
  11. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  12. 2020Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformerscited by 67
6 more
  1. 2023DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Modelscited by 36
  2. 2023Task-Agnostic Structured Pruning of Speech Representation Modelscited by 15
  3. 2023I3D: Transformer architectures with input-dependent dynamic depth for speech recognitioncited by 10
  4. 2024CULL-MT: Compression Using Language and Layer pruning for Machine Translationcited by 3
  5. 2016Neural Machine Translation from Simplified Translationscited by 2
  6. 2025IteRABRe: Iterative Recovery-Aided Block Reductioncited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025Iterative Layer Pruning for Efficient Translation Inferencecited by 4
  2. 2026AfriNLLB: Efficient Translation Models for African Languagescited by 2
  3. 2026One Voice, Many Tongues: Cross-Lingual Voice Cloning for Scientific Speechcited by 0
  4. 2026Text Scores Do Not Establish Performance on Lexically Non-Diagnostic Speech Tasks: A Qwen2-Audio Quantization Case Studycited by 0
  5. 2026Investigating Model Compression for Neural Machine Translation in the Biomedical Domaincited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026X-AuT: Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillation
  2. 2024Compact Language Models via Pruning and Knowledge Distillation
  3. 2025A Systematic Study of Compression Ordering for Large Language Models
  4. 2023CoLLD: Contrastive Layer-to-layer Distillation for Compressing Multilingual Pre-trained Speech Encoders
  5. 2025Novel Parasitic Dual-Scale Modeling for Efficient and Accurate Multilingual Speech Translation
  6. 2022AlphaTuning: Quantization-Aware Parameter-Efficient Adaptation of Large-Scale Pre-Trained Language Models
  7. 2024Model Compression and Efficient Inference for Large Language Models: A Survey
  8. 2023A Survey on Model Compression for Large Language Models