# QUADS: QUAntized Distillation Framework for Efficient Speech Language Understanding

arXiv: https://arxiv.org/abs/2505.14723 · HTML: https://mlsanity.com/paper/2505.14723/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 33 references that are in the map.
- First posted on arXiv 19 May 2025.

## Builds on (most cited first, 17 of 17)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Quantized Convolutional Neural Networks for Mobile Devices](https://mlsanity.com/paper/1512.06473/) (2015)
- [SLURP: A Spoken Language Understanding Resource Package](https://mlsanity.com/paper/2011.13205/) (2020)
- [Speech Model Pre-training for End-to-End Spoken Language Understanding](https://mlsanity.com/paper/1904.03670/) (2019)
- [TernaryBERT: Distillation-aware Ultra-low Bit BERT](https://mlsanity.com/paper/2009.12812/) (2020)
- [Pruning from Scratch](https://mlsanity.com/paper/1909.12579/) (2019)
- [DQ-BART: Efficient Sequence-to-Sequence Model via Joint Distillation and Quantization](https://mlsanity.com/paper/2203.11239/) (2022)
- [DQ-Whisper: Joint Distillation and Quantization for Efficient Multilingual Speech Recognition](https://mlsanity.com/paper/2305.10788/) (2023)
- [ML-LMCL: Mutual Learning and Large-Margin Contrastive Learning for Improving ASR Robustness in Spoken Language Understanding](https://mlsanity.com/paper/2311.11375/) (2023)
- [A neural prosody encoder for end-ro-end dialogue act classification](https://mlsanity.com/paper/2205.05590/) (2022)
- [Improving End-to-End SLU performance with Prosodic Attention and Distillation](https://mlsanity.com/paper/2305.08067/) (2023)
- [Speech Understanding on Tiny Devices with A Learning Cache](https://mlsanity.com/paper/2311.18188/) (2023)
- [LANID: LLM-assisted New Intent Discovery](https://mlsanity.com/paper/2503.23740/) (2025)
- [Large Language Models for Expansion of Spoken Language Understanding Systems to New Languages](https://mlsanity.com/paper/2404.02588/) (2024)

## Built on by (most cited first, 2 of 2)

- [Toward Low-Latency End-to-End Voice Agents for Telecommunications Using Streaming ASR, Quantized LLMs, and Real-Time TTS](https://mlsanity.com/paper/2508.04721/) (2025)
- [Mindfulness Meditation and Respiration: Accelerometer-Based Respiration Rate and Mindfulness Progress Estimation to Enhance App Engagement and Mindfulness Skills](https://mlsanity.com/paper/2507.17688/) (2025)

## Abstract (arXiv)

Spoken Language Understanding (SLU) systems must balance performance and efficiency, particularly in resource-constrained environments. Existing methods apply distillation and quantization separately, leading to suboptimal compression as distillation ignores quantization constraints. We propose QUADS, a unified framework that optimizes both through multi-stage training with a pre-tuned model, enhancing adaptability to low-bit regimes while maintaining accuracy. QUADS achieves 71.13% accuracy on SLURP and 99.20% on FSC, with only minor degradations of up to 5.56% compared to state-of-the-art models. Additionally, it reduces computational complexity by 60--73× (GMACs) and model size by 83--700×, demonstrating strong robustness under extreme quantization. These results establish QUADS as a highly efficient solution for real-world, resource-constrained SLU applications.