# Gaming the Metric, Not the Harm: Certifying Safety Audits against Strategic Platform Manipulation

arXiv: https://arxiv.org/abs/2605.06324 · HTML: https://mlsanity.com/paper/2605.06324/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 33 references that are in the map.
- First posted on arXiv 7 May 2026.

## Builds on (most cited first, 13 of 13)

- [Performative Prediction](https://mlsanity.com/paper/2002.06673/) (2020)
- [Strategic Classification](https://mlsanity.com/paper/1506.06980/) (2015)
- [HateCheck: Functional Tests for Hate Speech Detection Models](https://mlsanity.com/paper/2012.15606/) (2020)
- [Defining and Characterizing Reward Hacking](https://mlsanity.com/paper/2209.13085/) (2022)
- [Evaluations of Machine Learning Privacy Defenses are Misleading](https://mlsanity.com/paper/2404.17399/) (2024)
- [Membership Inference Attacks as Privacy Tools: Reliability, Disparity and Ensemble](https://mlsanity.com/paper/2506.13972/) (2025)
- [The Accountability Paradox: How Platform API Restrictions Undermine AI Transparency Mandates](https://mlsanity.com/paper/2505.11577/) (2025)
- [The Amplification Paradox in Recommender Systems](https://mlsanity.com/paper/2302.11225/) (2023)
- [Harm Mitigation in Recommender Systems under User Preference Dynamics](https://mlsanity.com/paper/2406.09882/) (2024)
- [Auditing Recommender Systems -- Putting the DSA into practice with a risk-scenario-based approach](https://mlsanity.com/paper/2302.04556/) (2023)
- [You Don't Bring Me Flowers: Mitigating Unwanted Recommendations Through Conformal Risk Control](https://mlsanity.com/paper/2507.16829/) (2025)
- [With a Little Help From My Friends: Collective Manipulation in Risk-Controlling Recommender Systems](https://mlsanity.com/paper/2603.28476/) (2026)
- [A Unified Causal Framework for Auditing Recommender Systems for Ethical Concerns](https://mlsanity.com/paper/2409.13210/) (2024)

## Abstract (arXiv)

Online-safety regulation under the UK Online Safety Act and the EU Digital Services Act increasingly treats scalar metrics as compliance evidence. Once announced, such a metric also becomes an optimization target: a strategic platform can improve its score by routing recommendations through semantically equivalent content variants, without reducing true harm. We ask when such an audit metric can still certify a genuine reduction in harm. The protocol is modeled as a published transformation graph whose connected components form semantic classes, and the metric itself is treated as a security object. Three results follow. First, any metric that scores variants directly is manipulable as soon as two equivalent variants in a harmful class disagree in score. Second, the semantic-envelope lift, which assigns each variant the maximum score in its class, is the unique pointwise minimum among conservative classwise-constant repairs. Third, a class-stratified certificate, H^star(x) ≤ (1/hatα) MEnv(m)(x) + barη, holds for every platform strategy, with barη absorbing annotation and protocol error. We check the claims at three levels: exhaustive enumeration on a finite-state grid of mixed strategies, an SMT encoding in Z3 cross-replayed in cvc5, and a bounded single-player MDP encoded in PRISM-games. The fragile metric fails manipulation invariance and cannot support the same useful predeclared class-coverage certificate; under the envelope-level certificate, it produces large violations at every tested instance, with a large mean gaming gap across random catalogs at a fixed audit budget. The semantic-envelope metric exhibits no such violation in the tested instances.