Ensemble Monitoring for AI Control: Diverse Signals Outweigh More Compute
On the map: LLM agents and tool use → Agent threats, guardrails and governance
Abstract (arXiv)
As AI systems are increasingly deployed in autonomous agentic settings at scale, it is important to ensure the actions they take are safe and aligned with user intent. Monitoring agent actions is a key safety mechanism, yet reliable monitors remain difficult to build and the scale of these systems makes human oversight impractical. We show that combining signals from diverse monitors into an ensemble improves detection of misaligned actions. We build 12 GPT-4.1-Mini monitors using both prompting and fine-tuning strategies. We evaluate them on coding tasks where candidate solutions pass standard tests but fail on adversarial inputs. In this setting, diverse ensembles outperform both individual monitors and homogeneous ensembles. Our best 3-monitor ensemble achieves 2.4x greater detection performance gain compared to an ensemble composed of three identical monitors, with the same ensemble performing strongly on an independent dataset. We contend that these results show that diversity - not scale - drives gains. The best ensembles combine strong individual performance with low correlation between monitors. Furthermore, fine-tuned monitors appear in every top-performing ensemble and maintain this advantage on out-of-distribution attack types, suggesting that fine-tuning enables detection capabilities that prompting alone does not elicit. These results support ensemble monitoring as a practical AI control strategy for safety gains at reasonable inference costs.
Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 13 of its 32 references that are in the map. First posted on arXiv 14 May 2026.
Builds on
13 of 32 references.
Closest ancestors · 4
Same direction, newest first
- 2025Async Control: Stress-testing Asynchronous Control Measures for LLM Agents15 Dec
- 2025Adaptive Attacks on Trusted Monitors Subvert AI Control Protocols10 Oct
- 2025Control Tax: The Price of Keeping AI in Check5 Jun
- 2025How to evaluate control measures for LLM agents? A trajectory from today to superintelligence7 Apr
Other references · 9
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2021Measuring Coding Challenge Competence With APPScited by 902
- 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
- 2018AI safety via debatecited by 400
- 2025Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safetycited by 249
- 2023AI Control: Improving Safety Despite Intentional Subversioncited by 229
- 2025Detecting Strategic Deception Using Linear Probescited by 68
- 2025Ctrl-Z: Controlling AI Agents via Resamplingcited by 44
- 2025Combining Cost-Constrained Runtime Monitors for AI Safetycited by 10
Built on by
Citing papers per year: 2026: 6.
6 papers.
- 2026Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineagescited by 1
- 2026The Horizon Gap: Planning, Memory, Execution, Training, and Evaluation for Long-Horizon LLM Agentscited by 1
- 2026Reliability Theory for AI Controlcited by 1
- 2026Decorrelation Is Not Complementarity: Skill, Not Lineage, Governs Trusted-Monitor Ensemblescited by 0
- 2026Improving scalable oversight with co-trained monitorscited by 0
- 2026Towards a Unified Misuse Monitoring Benchmarkcited by 0
- 2026Towards a Unified Misuse Monitoring Benchmark5 Oct 2026
- 2026Improving scalable oversight with co-trained monitors28 Sep 2026
- 2026Reliability Theory for AI Control22 Sep 2026
- 2026Decorrelation Is Not Complementarity: Skill, Not Lineage, Governs Trusted-Monitor Ensembles17 Aug 2026
- 2026The Horizon Gap: Planning, Memory, Execution, Training, and Evaluation for Long-Horizon LLM Agents7 Aug 2026
- 2026Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages6 Jul 2026
Similar, not cited
Close in wording, no citation either way
- 2026Aligned Alone, Misaligned Together: Forecasting Adversarial Capture in LLM Agent Populations
- 2026Training Deliberative Monitors for Black-Box Scheming Detection
- 2025Reliable Weak-to-Strong Monitoring of LLM Agents
- 2026MonitoringBench: Semi-Automated Red-Teaming for Agent Monitoring
- 2024Learning Ensembles of Vision-based Safety Control Filters
- 2024The Curse of Diversity in Ensemble-Based Exploration
- 2026Red-Teaming Auto Mode: Improving Blocking Classifiers Against Malign Coding Agents
- 2025An Outlook on the Opportunities and Challenges of Multi-Agent AI Systems