# How Hackable Is Your Speech Quality Metric? A Corrected Protocol, a Benchmark, and What Patching Buys

arXiv: https://arxiv.org/abs/2610.10899 · HTML: https://mlsanity.com/paper/2610.10899/

## Key facts

- No citing papers in the map yet (data through 7 Oct 2026).
- Builds on 10 of its 38 references that are in the map.
- First posted on arXiv 7 Oct 2026.

## Builds on (most cited first, 10 of 10)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Adversarial Training of Reward Models](https://mlsanity.com/paper/2504.06141/) (2025)
- [Reward Model Overoptimisation in Iterated RLHF](https://mlsanity.com/paper/2505.18126/) (2025)
- [Hardening Agent Benchmarks with Adversarial Hacker-Fixer Loops](https://mlsanity.com/paper/2606.08960/) (2026)
- [Cooper: Co-Optimizing Policy and Reward Models in Reinforcement Learning for Large Language Models](https://mlsanity.com/paper/2508.05613/) (2025)
- [On the robustness of non-intrusive speech quality model by adversarial examples](https://mlsanity.com/paper/2211.06508/) (2022)
- [EventTrojan: Manipulating Non-Intrusive Speech Quality Assessment via Imperceptible Events](https://mlsanity.com/paper/2309.01480/) (2023)
- [UTDUSS: UTokyo-SaruLab System for Interspeech2024 Speech Processing Using Discrete Speech Unit Challenge](https://mlsanity.com/paper/2403.13720/) (2024)
- [Attacking UTMOS: Probing the Robustness of a Speech Quality Assessment Model](https://mlsanity.com/paper/2606.31105/) (2026)
- [The Limits of Reference-Free Speech Quality Metrics as Evaluators and Rewards on Modern Text-to-Speech](https://mlsanity.com/paper/2609.13150/) (2026)

## Abstract (arXiv)

Speech quality predictors are increasingly used as rewards, yet no agreed measure of their hackability exists. The usual measurement has two flaws. First, the perturbation reaches the predictor through a processing chain -- here a neural codec -- that shifts the score on its own, which scoring against the raw input charges to the attack. Referencing the unperturbed round trip instead changes measured hackability by up to a factor of four (0.31 to 0.08 for one defence). Second, one trained attacker is a sample, not a measurement: five attackers differing only in random seed reach success rates from 0.00 to 0.38 against one fixed predictor, so a defence claim needs the worst case over several. Under this protocol, four published predictors differ widely: NISQA is hacked on 90% of utterances, SSL-MOS on 21%, DNSMOS on 14% and UTMOS on 6%. We then audit a closed attack-detect-patch loop. It hardens the predictor only in its own attack space, by less than the spread between attackers; a random-perturbation baseline matches it; and it costs up to 0.30 system SRCC out of domain. Enhancers post-trained against patched predictors hack them far less (PESQ -0.03 versus -0.23). Code, preregistration and run outputs are released.