mlsanityData to 14 Oct 2026

Hard-Gate Candidacy in a Deployed Validator Suite

arXiv 2609.39037 ↗First posted 30 Sep 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Before a validator can be promoted to a hard gate on a deployment pipeline, it has to be shown that its firing separates outputs that reach users in working order from those that do not. We run that screen on 13 validators in a deployed generative agent, against 550 runtime and 350 static builds labelled by downstream outcome, and report each check's marginal separation J=TPR-FPR with Newcombe intervals and Fisher exact tests. Two checks survive correction for multiple comparisons, two more are nominal only, and the remaining nine are not distinguishable from zero, three of them because they never fired on any sampled build. Execution itself is not random with respect to the property being gated, and this replicates: across four runs covering 1,867 builds and ten distinct runtime checks, probes were skipped on 144 of 895 broken builds and 1 of 972 acceptable builds (per-run rates 15.6% to 16.6% against at most 0.3%), every skip carrying the same unsafe-to-probe reason. Because a skipped check is recorded as a pass, this imposes a ceiling that no check quality can lift: a check that needs a live artifact cannot operationally detect more than about 84% of broken builds in this harness. For the one check with construct-specific labels, a detector built for blank output fires on 0 of 90 human-labelled blank builds (95% upper bound on sensitivity 3.3%), and the global frame statistic it approximates separates the classes only weakly (AUC 0.59), so the gap is not a threshold that needs tuning. The same gap appears one layer up: on a census of tens of thousands of judge-scored builds, 32.5% of rejections carry no recorded issue at all. We argue that evaluation records must distinguish a check that ran and passed from one that did not run, must carry the evidence for a rejection, and that an inventory of checks is not evidence about a gate.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 13 of its 21 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

13 of 21 references.

References · 13

  1. 2017Selective Classification for Deep Neural Networkscited by 852
  2. 2024Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreementcited by 105
  3. 2024Evaluating LLMs at Detecting Errors in LLM Responsescited by 50
  4. 2024Open Problems in Technical AI Governancecited by 49
  5. 2025Lost in Moderation: How Commercial Content Moderation APIs Over- and Under-Moderate Group-Targeted Hate Speech and Linguistic Variationscited by 13
  6. 2022Reliable Decision from Multiple Subtasks through Threshold Optimization: Content Moderation in the Wildcited by 7
  7. 2026SWE-Cycle: Benchmarking Code Agents across the Complete Issue Resolution Cyclecited by 5
  8. 2026Catching One in Five: LLM-as-Judge Blind Spots in Production Multi-Turn Transaction Agentscited by 4
  9. 2016Will My Tests Tell Me If I Break This Code?cited by 2
  10. 2026When Errors Become Narratives: A Longitudinal Taxonomy of Silent Failures in a Production LLM Agent Runtimecited by 2
  11. 2026HALLMARK: How to Diagnose LLM Citation Verifiers and When to Deploy Themcited by 2
  12. 2018A Comprehensive Study of Pseudo-tested Methodscited by 1
1 more
  1. 2026BELLS-O: Evaluating the Operational Trade-offs of LLM Supervision Systemscited by 1

Built on by

Not cited yet.