# Scaling Audio Models Efficiently: Joint Optimization of Scale, Resolution, Adaptation, Precision, and Sparsity

arXiv: https://arxiv.org/abs/2606.22790 · HTML: https://mlsanity.com/paper/2606.22790/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 11 references that are in the map.
- First posted on arXiv 22 Jun 2026.

## Builds on (most cited first, 7 of 7)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [USM-Lite: Quantization and Sparsity Aware Fine-tuning for Speech Recognition with Universal Speech Models](https://mlsanity.com/paper/2312.08553/) (2023)
- [Folding Attention: Memory and Power Optimization for On-Device Transformer-based Streaming Speech Recognition](https://mlsanity.com/paper/2309.07988/) (2023)
- [Effective and Efficient Mixed Precision Quantization of Speech Foundation Models](https://mlsanity.com/paper/2501.03643/) (2025)

## Abstract (arXiv)

Large automatic speech recognition (ASR) models such as Whisper must be deployed across hardware with widely varying memory and inference-speed constraints. We present a compression framework that jointly parametrizes Whisper deployment along six dimensions: model size, temporal resolution, encoder token stride, low-rank adaptation capacity, weight precision and sparsity pattern. All axes are jointly optimized using NSGA-III with respect to three deployment objectives: word error rate (WER), inference FLOPs, and memory footprint. Across 50 of the 1,680 candidate configurations evaluated, we characterize the conditional effect of each axis and identify compression combinations that dominate naive single-axis scaling, while finding that 1:4 structured sparsity fails to recover acceptable accuracy under the tested recovery budgets. We report measured WER and resident memory, use analytical EffFLOPs as the search-time compute surrogate, and separately validate representative inference configurations using measured real-time factor (RTF).