mlsanityData to 7 Oct 2026

Gaming the Metric, Not the Harm: Certifying Safety Audits against Strategic Platform Manipulation

arXiv 2605.06324 ↗First posted 7 May 2026cs.CR cs.CY cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Online-safety regulation under the UK Online Safety Act and the EU Digital Services Act increasingly treats scalar metrics as compliance evidence. Once announced, such a metric also becomes an optimization target: a strategic platform can improve its score by routing recommendations through semantically equivalent content variants, without reducing true harm. We ask when such an audit metric can still certify a genuine reduction in harm. The protocol is modeled as a published transformation graph whose connected components form semantic classes, and the metric itself is treated as a security object. Three results follow. First, any metric that scores variants directly is manipulable as soon as two equivalent variants in a harmful class disagree in score. Second, the semantic-envelope lift, which assigns each variant the maximum score in its class, is the unique pointwise minimum among conservative classwise-constant repairs. Third, a class-stratified certificate, H^star(x) ≤ (1/hatα) MEnv(m)(x) + barη, holds for every platform strategy, with barη absorbing annotation and protocol error. We check the claims at three levels: exhaustive enumeration on a finite-state grid of mixed strategies, an SMT encoding in Z3 cross-replayed in cvc5, and a bounded single-player MDP encoded in PRISM-games. The fragile metric fails manipulation invariance and cannot support the same useful predeclared class-coverage certificate; under the envelope-level certificate, it produces large violations at every tested instance, with a large mean gaming gap across random catalogs at a fixed audit budget. The semantic-envelope metric exhibits no such violation in the tested instances.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 13 of its 33 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

13 of 33 references.

References · 13

  1. 2020Performative Predictioncited by 318
  2. 2015Strategic Classificationcited by 256
  3. 2020HateCheck: Functional Tests for Hate Speech Detection Modelscited by 215
  4. 2022Defining and Characterizing Reward Hackingcited by 184
  5. 2024Evaluations of Machine Learning Privacy Defenses are Misleadingcited by 35
  6. 2025Membership Inference Attacks as Privacy Tools: Reliability, Disparity and Ensemblecited by 11
  7. 2025The Accountability Paradox: How Platform API Restrictions Undermine AI Transparency Mandatescited by 7
  8. 2023The Amplification Paradox in Recommender Systemscited by 6
  9. 2024Harm Mitigation in Recommender Systems under User Preference Dynamicscited by 5
  10. 2023Auditing Recommender Systems -- Putting the DSA into practice with a risk-scenario-based approachcited by 4
  11. 2025You Don't Bring Me Flowers: Mitigating Unwanted Recommendations Through Conformal Risk Controlcited by 3
  12. 2026With a Little Help From My Friends: Collective Manipulation in Risk-Controlling Recommender Systemscited by 2
1 more
  1. 2024A Unified Causal Framework for Auditing Recommender Systems for Ethical Concernscited by 1

Built on by

Not cited yet.