# Core Safety Values for Provably Corrigible Agents

arXiv: https://arxiv.org/abs/2507.20964 · HTML: https://mlsanity.com/paper/2507.20964/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 14 of its 40 references that are in the map.
- First posted on arXiv 28 Jul 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward hacking and misspecification](https://mlsanity.com/alignment/reward-hacking-and-misspecification/)

## Builds on (most cited first, 14 of 14)

- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [AI safety via debate](https://mlsanity.com/paper/1805.00899/) (2018)
- [Supervising strong learners by amplifying weak experts](https://mlsanity.com/paper/1810.08575/) (2018)
- [Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective](https://mlsanity.com/paper/1908.04734/) (2019)
- [The Off-Switch Game](https://mlsanity.com/paper/1611.08219/) (2016)
- [Scalable AI Safety via Doubly-Efficient Debate](https://mlsanity.com/paper/2311.14125/) (2023)
- [Penalizing side effects using stepwise relative reachability](https://mlsanity.com/paper/1806.01186/) (2018)
- [Conservative Agency via Attainable Utility Preservation](https://mlsanity.com/paper/1902.09725/) (2019)
- [Avoiding Obfuscation with Prover-Estimator Debate](https://mlsanity.com/paper/2506.13609/) (2025)
- [A Game-Theoretic Analysis of the Off-Switch Game](https://mlsanity.com/paper/1708.03871/) (2017)
- [The Partially Observable Off-Switch Game](https://mlsanity.com/paper/2411.17749/) (2024)
- [Intrinsic Barriers and Practical Pathways for Human-AI Alignment: An Agreement-Based Complexity Analysis](https://mlsanity.com/paper/2502.05934/) (2025)
- [Reachability Analysis of Randomly Perturbed Hamiltonian Systems](https://mlsanity.com/paper/2105.01942/) (2021)

## Built on by (most cited first, 5 of 5)

- [The Oversight Game: Learning to Cooperatively Balance an AI Agent's Safety and Autonomy](https://mlsanity.com/paper/2510.26752/) (2025)
- [ROGUE: Evaluating Corrigibility Failures in Frontier Computer-Use Agents](https://mlsanity.com/paper/2606.00341/) (2026)
- [The Digital Apprentice: A Framework for Human-Directed Agentic AI Development](https://mlsanity.com/paper/2606.04321/) (2026)
- [Can We Stop Malicious AI? KILLBENCH: A Benchmark for External AI Kill Switch Feasibility](https://mlsanity.com/paper/2511.13725/) (2025)
- [KYA: A Framework-Agnostic Trust Layer for Autonomous Systems with Verifiable Provenance and Hierarchical Policy Composition](https://mlsanity.com/paper/2605.25376/) (2026)

## Abstract (arXiv)

We introduce the first complete formal solution to corrigibility in the off-switch game, with provable guarantees in multi-step, partially observed environments. Our framework consists of five *structurally separate* utility heads -- deference, switch-access preservation, truthfulness, low-impact behavior via a belief-based extension of Attainable Utility Preservation, and bounded task reward -- combined lexicographically by strict weight gaps. Theorem 1 proves exact single-round corrigibility in the partially observable off-switch game; Theorem 3 extends the guarantee to multi-step, self-spawning agents, showing that even if each head is *learned* to mean-squared error varepsilon and the planner is varepsilon-sub-optimal, the probability of violating *any* safety property is bounded while still ensuring net human benefit. In contrast to Constitutional AI or RLHF/RLAIF, which merge all norms into one learned scalar, our separation makes obedience and impact-limits provably dominate even when incentives conflict. For settings where adversaries can modify the agent, we prove that deciding whether an arbitrary post-hack agent will ever violate corrigibility is undecidable by reduction to the halting problem, then carve out a finite-horizon "decidable island" where safety can be certified in randomized polynomial time and verified with privacy-preserving, constant-round zero-knowledge proofs.