# Text Scores Do Not Establish Performance on Lexically Non-Diagnostic Speech Tasks: A Qwen2-Audio Quantization Case Study

arXiv: https://arxiv.org/abs/2609.26823 · HTML: https://mlsanity.com/paper/2609.26823/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 18 references that are in the map.
- First posted on arXiv 20 Sep 2026.

## Builds on (most cited first, 11 of 11)

- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](https://mlsanity.com/paper/2205.12446/) (2022)
- [Do Audio LLMs Really LISTEN, or Just Transcribe? Measuring Lexical vs. Acoustic Emotion Cues Reliance](https://mlsanity.com/paper/2510.10444/) (2025)
- [Efficient Speech Translation through Model Compression and Knowledge Distillation](https://mlsanity.com/paper/2505.20237/) (2025)
- [A Closer Look at Failure Modes in Temporal Understanding of Large Audio-Language Models](https://mlsanity.com/paper/2606.17417/) (2026)
- [From Semantics to Readout: Mechanistic Understanding of Audio Tokens after Fine-Tuning for Temporal Audio Grounding](https://mlsanity.com/paper/2607.25355/) (2026)
- [Direct or Mediated? Task-Dependent Audio Information Routing in Large Audio Language Models](https://mlsanity.com/paper/2608.27026/) (2026)
- [TEMPO: Temporally-grounded Multi-task Post-training for Large Audio-Language Models](https://mlsanity.com/paper/2608.29999/) (2026)
- [TAG-Bench: Benchmarking Temporal Audio Grounding in Large Audio Language Models](https://mlsanity.com/paper/2609.01542/) (2026)

## Abstract (arXiv)

Text-output scores alone do not show whether quantization preserves performance on speech tasks whose target labels cannot be recovered from the transcript. We evaluate fixed mixed 4/8-bit Qwen2-Audio-7B-Instruct allocations averaging 6 and 7 bits per parameter on 508 English-to-German FLEURS utterances and on 512 RAVDESS emotion clips from 16 speakers. The BLEU and chrF differences from half precision (FP16) have intervals that include zero for both allocations. On RAVDESS, the same two sentences occur equally often with every emotion label. The absolute accuracy differences from FP16 are -3.71% for 6 bit and -1.17% for 7 bit. The 6-bit speaker interval excludes zero and an exact two-sided sign-flip test gives p=0.0148; the 7-bit interval includes zero. Same-budget controls do not identify either selected allocation as best. This case study shows why translation scores and performance on tasks beyond the transcript need separate evaluation.