# X-AuT: Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillation

arXiv: https://arxiv.org/abs/2609.11412 · HTML: https://mlsanity.com/paper/2609.11412/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation.
- Builds on 19 of its 25 references that are in the map.
- First posted on arXiv 10 Sep 2026.

## Placed in

- Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/)

## Builds on (most cited first, 19 of 19)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](https://mlsanity.com/paper/2205.12446/) (2022)
- [Reducing Transformer Depth on Demand with Structured Dropout](https://mlsanity.com/paper/1909.11556/) (2019)
- [GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio](https://mlsanity.com/paper/2106.06909/) (2021)
- [AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline](https://mlsanity.com/paper/1709.05522/) (2017)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [A Survey of On-Policy Distillation for Large Language Models](https://mlsanity.com/paper/2604.00626/) (2026)
- [Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling](https://mlsanity.com/paper/2311.00430/) (2023)
- [An Embarrassingly Simple Approach for LLM with Strong ASR Capacity](https://mlsanity.com/paper/2402.08846/) (2024)
- [THCHS-30 : A Free Chinese Speech Corpus](https://mlsanity.com/paper/1512.01882/) (2015)
- [Dynamic Encoder Size Based on Data-Driven Layer-wise Pruning for Speech Recognition](https://mlsanity.com/paper/2407.18930/) (2024)
- [LiteASR: Efficient Automatic Speech Recognition with Low-Rank Approximation](https://mlsanity.com/paper/2502.20583/) (2025)
- [Does the Definition of Difficulty Matter? Scoring Functions and their Role for Curriculum Learning](https://mlsanity.com/paper/2411.00973/) (2024)
- [On the Role of Encoder Depth: Pruning Whisper and LoRA Fine-Tuning in SLAM-ASR](https://mlsanity.com/paper/2603.27981/) (2026)
- [Structured Sparsity and Weight-adaptive Pruning for Memory and Compute efficient Whisper models](https://mlsanity.com/paper/2510.12666/) (2025)
- [ASKD-Whisper: Adaptive Self-knowledge Distillation for Efficient and Low-Latency Automatic Speech Recognition](https://mlsanity.com/paper/2601.19919/) (2026)
- [Data-Efficient On-Policy Distillation for Automatic Speech Recognition](https://mlsanity.com/paper/2605.28139/) (2026)

## Abstract (arXiv)

Reducing audio-encoder depth lowers the inference cost of speech large language models, but removing complete blocks perturbs the embeddings consumed by the decoder and can cause deletion and premature end-of-sequence errors. We introduce X-AuT, a progressive framework that selects layer combinations through short behavioral probes and restores the pruned model through representation alignment, cross-scale distillation, scheduled student-policy supervision, and LoRA finetuning. The language-model backbone remains frozen, while attention LoRA adapters and the tied output embedding adapt during distillation. Training uses the highest-agreement tier from a transcript-consistency pipeline, followed by source reweighting during finetuning. On ten public Chinese--English benchmarks, compressing Qwen3-ASR-0.6B from 18 to 16 audio-encoder layers reduces macro-average error from 5.61% to 5.27%. The 14-layer model reaches 5.75% with 20.7% fewer audio-tower parameters. Under the matched recipe, the 1.7B teacher yields 5.55% mean error, compared with 8.45% for self-distillation, and progressive 18rightarrow14 pruning outperforms direct pruning (5.75% vs. 6.73%). These single-run results establish two practical operating points and show that the accuracy effects vary across benchmarks. Project website: https://xpeng-ai.github.io/x-aut