# Code Monitor Red Teaming for Public-Test-Passing Code

arXiv: https://arxiv.org/abs/2607.20852 · HTML: https://mlsanity.com/paper/2607.20852/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security.
- Builds on 26 of its 30 references that are in the map.
- First posted on arXiv 23 Jul 2026.

## Placed in

- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 26 of 26)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision](https://mlsanity.com/paper/2312.09390/) (2023)
- [DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation](https://mlsanity.com/paper/2211.11501/) (2022)
- [CodeT: Code Generation with Generated Tests](https://mlsanity.com/paper/2207.10397/) (2022)
- [CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution](https://mlsanity.com/paper/2401.03065/) (2024)
- [LEVER: Learning to Verify Language-to-Code Generation with Execution](https://mlsanity.com/paper/2302.08468/) (2023)
- [AI Control: Improving Safety Despite Intentional Subversion](https://mlsanity.com/paper/2312.06942/) (2023)
- [LLM Critics Help Catch LLM Bugs](https://mlsanity.com/paper/2407.00215/) (2024)
- [SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents](https://mlsanity.com/paper/2406.12952/) (2024)
- [On scalable oversight with weak LLMs judging strong LLMs](https://mlsanity.com/paper/2407.04622/) (2024)
- [CriticBench: Benchmarking LLMs for Critique-Correct Reasoning](https://mlsanity.com/paper/2402.14809/) (2024)
- [Prover-Verifier Games improve legibility of LLM outputs](https://mlsanity.com/paper/2407.13692/) (2024)
- [CoT Red-Handed: Stress Testing Chain-of-Thought Monitoring](https://mlsanity.com/paper/2505.23575/) (2025)
- [CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation](https://mlsanity.com/paper/2311.08588/) (2023)
- [Games for AI Control: Models of Safety Evaluations of AI Deployment Protocols](https://mlsanity.com/paper/2409.07985/) (2024)
- [Reliable Weak-to-Strong Monitoring of LLM Agents](https://mlsanity.com/paper/2508.19461/) (2025)
- [CodeEditorBench: Evaluating Code Editing Capability of Large Language Models](https://mlsanity.com/paper/2404.03543/) (2024)
- [Embodied Red Teaming for Auditing Robotic Foundation Models](https://mlsanity.com/paper/2411.18676/) (2024)
- [RedCoder: Automated Multi-Turn Red Teaming for Code LLMs](https://mlsanity.com/paper/2507.22063/) (2025)
- [MonitoringBench: Semi-Automated Red-Teaming for Agent Monitoring](https://mlsanity.com/paper/2605.09684/) (2026)

## Abstract (arXiv)

Visible tests are a common gate for LLM-generated code, but passing them does not certify specification correctness. We study a deployment-like monitoring problem: after code has passed public tests, can a weaker LLM verifier identify the residual hidden bugs? We introduce Code Monitor Red Teaming, a monitor-red-teaming protocol that fixes a public-check information boundary while varying generator pressure, verifier scaffolding, and weak-to-strong capability. We instantiate it as CodeMonitorBench, spanning function-level, data-science, and workflow code. Across 71,000 generated candidates, 43,677 pass public tests and 23,081 of those fail hidden tests. Weak verifiers improve with scaffolding and model family, but still miss most hidden bugs at 5% false-positive rate. As a robustness stress test, adversarial public-test-overfit pressure lowers verifier AUROC and raises low-FPR miss rates in most cells. A GLM-5.1 verifier recovers part of the gap under the same evidence boundary; an inferability audit shows that remaining misses mix verifier failures with M1 evidence limits.