# Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages

arXiv: https://arxiv.org/abs/2607.06596 · HTML: https://mlsanity.com/paper/2607.06596/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 18 references that are in the map.
- First posted on arXiv 6 Jul 2026.

## Builds on (most cited first, 16 of 16)

- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [AI Control: Improving Safety Despite Intentional Subversion](https://mlsanity.com/paper/2312.06942/) (2023)
- [SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents](https://mlsanity.com/paper/2506.15740/) (2025)
- [Reliable Weak-to-Strong Monitoring of LLM Agents](https://mlsanity.com/paper/2508.19461/) (2025)
- [Adaptive Attacks on Trusted Monitors Subvert AI Control Protocols](https://mlsanity.com/paper/2510.09462/) (2025)
- [Async Control: Stress-testing Asynchronous Control Measures for LLM Agents](https://mlsanity.com/paper/2512.13526/) (2025)
- [How does information access affect LLM monitors' ability to detect sabotage?](https://mlsanity.com/paper/2601.21112/) (2026)
- [Evaluating Control Protocols for Untrusted AI Agents](https://mlsanity.com/paper/2511.02997/) (2025)
- [Control Tax: The Price of Keeping AI in Check](https://mlsanity.com/paper/2506.05296/) (2025)
- [Practical challenges of control monitoring in frontier AI deployments](https://mlsanity.com/paper/2512.22154/) (2025)
- [Attack Selection Reduces Safety in Concentrated AI Control Settings against Trusted Monitoring](https://mlsanity.com/paper/2602.04930/) (2026)
- [Ensemble Monitoring for AI Control: Diverse Signals Outweigh More Compute](https://mlsanity.com/paper/2605.15377/) (2026)
- [CIAware-Bench: Benchmarking Control Intervention Awareness Across Frontier LLMs](https://mlsanity.com/paper/2606.11063/) (2026)
- [Classifier Context Rot: Monitor Performance Degrades with Context Length](https://mlsanity.com/paper/2605.12366/) (2026)
- [Cross-LLM Generalization of Behavioral Backdoor Detection in AI Agent Supply Chains](https://mlsanity.com/paper/2511.19874/) (2025)

## Built on by (most cited first, 1 of 1)

- [Decorrelation Is Not Complementarity: Skill, Not Lineage, Governs Trusted-Monitor Ensembles](https://mlsanity.com/paper/2608.16190/) (2026)

## Abstract (arXiv)

Trusted monitoring is a central defense in AI control: a cheaper trusted model scores an untrusted model's actions for sabotage, and the most suspicious are audited or deferred. Such monitors are evaluated against one or two untrusted models, and the accuracy is reported as a property of the monitor. We ask whether it is partly a property of the pairing. We make the untrusted policy family the controlled axis: we fit a monitor on family A's transcripts, apply it to family B, and decompose the cross-family AUROC into how obvious each family's sabotage is, how capable each monitor is, and the residual own-family advantage after both are removed: the interaction. On code-backdoor transcripts the interaction is positive and survives the dominant confounds: +0.172 (95% CI [+0.158, +0.185]) on four open-weight families on a strict leak-free basis, holding across an eight-family matrix spanning four frontier vendors, blatancy-matched controls, a trained transcript-reading monitor up to 34B parameters, and a second code benchmark. The gap is geometric: each lineage's sabotage occupies a different scoring direction, and rotating the monitor's scoring direction toward the target lineage recovers most of the lost detection while a matched random rotation recovers nothing. At a 1% audit budget under defer-to-trusted, an off-lineage monitor catches 19% of attack attempts where an in-lineage monitor catches 41% at the same audit cost, a +21.7-point gap. Control evaluations should report cross-family transfer matrices: a single-pairing accuracy overstates the safety a monitor delivers against a lineage it was not calibrated on. We give a four-step protocol that acts on the gap, with each step a measured result.