Information-Theoretic Limits of Safety Verification for Self-Improving Systems
Not on the map. Why?
Abstract (arXiv)
Can a safety gate permit unbounded beneficial self-modification while maintaining bounded cumulative risk? We formalize this question through dual conditions -- requiring sum delta_n < infinity (bounded risk) and sum TPR_n = infinity (unbounded utility) -- and establish a theory of their (in)compatibility. Classification impossibility (Theorem 1): For power-law risk schedules delta_n = O(n^{-p}) with p > 1, any classifier-based gate under overlapping safe/unsafe distributions satisfies TPR_n <= C_alpha * delta_n^beta via Holder's inequality, forcing sum TPR_n < infinity. This impossibility is exponent-optimal (Theorem 3). A second independent proof via the NP counting method (Theorem 4) yields a 13% tighter bound without Holder's inequality. Universal finite-horizon ceiling (Theorem 5): For any summable risk schedule, the exact maximum achievable classifier utility is U*(N, B) = N * TPR_NP(B/N), growing as exp(O(sqrt(log N))) -- subpolynomial. At N = 10^6 with budget B = 1.0, a classifier extracts at most U* ~ 87 versus a verifier's ~500,000. Verification escape (Theorem 2): A Lipschitz ball verifier achieves delta = 0 with TPR > 0, escaping the impossibility. Formal Lipschitz bounds for pre-LayerNorm transformers under LoRA enable LLM-scale verification. The separation is strict. We validate on GPT-2 (d_LoRA = 147,456): conditional delta = 0 with TPR = 0.352. Comprehensive empirical validation is in the companion paper [D2].
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 10 of its 39 references that are in the map. First posted on arXiv 30 Mar 2026.
Builds on
10 of 39 references.
References · 10
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2016Concrete Problems in AI Safetycited by 1,582
- 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
- 2018Robustness May Be at Odds with Accuracycited by 873
- 2018Scalable agent alignment via reward modeling: a research directioncited by 438
- 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 413
- 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 347
- 2019Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networkscited by 290
- 2020The Lipschitz Constant of Self-Attentioncited by 171
- 2021Lipschitz Normalization for Self-Attention Layers with Application to Graph Neural Networkscited by 38
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Safety Hacking in Constrained Best-of-N Inference-time Scaling
- 2026Bounded Reachability & Jailbreak Detection via Contraction-Constrained State Space Models
- 2026Falsifiable Release Gates for Self-Improving Systems: Standing Invariants at Scale
- 2025Verifiable Safety Q-Filters via Hamilton-Jacobi Reachability and Multiplicative Q-Networks
- 2026The Geometry of Refusal: Why Post-Hoc Safety Is Fragile and Pretraining-Time Safety Persists
- 2026Lipschitz-Based Robustness Certification Under Floating-Point Execution
- 2026Available Guardrails: Certifying Selective Prediction across ML Systems
- 2026The Architecture of Errors: From Universal Impossibility to Patch-Local LLM Reliability