mlsanityData to 7 Oct 2026

Code Monitor Red Teaming for Public-Test-Passing Code

arXiv 2607.20852 ↗First posted 23 Jul 2026cs.AIcited by 0

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Visible tests are a common gate for LLM-generated code, but passing them does not certify specification correctness. We study a deployment-like monitoring problem: after code has passed public tests, can a weaker LLM verifier identify the residual hidden bugs? We introduce Code Monitor Red Teaming, a monitor-red-teaming protocol that fixes a public-check information boundary while varying generator pressure, verifier scaffolding, and weak-to-strong capability. We instantiate it as CodeMonitorBench, spanning function-level, data-science, and workflow code. Across 71,000 generated candidates, 43,677 pass public tests and 23,081 of those fail hidden tests. Weak verifiers improve with scaffolding and model family, but still miss most hidden bugs at 5% false-positive rate. As a robustness stress test, adversarial public-test-overfit pressure lowers verifier AUROC and raises low-FPR miss rates in most cells. A GLM-5.1 verifier recovers part of the gap under the same evidence boundary; an inferability audit shows that remaining misses mix verifier failures with M1 evidence limits.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 26 of its 30 references that are in the map. First posted on arXiv 23 Jul 2026.

Builds on

26 of 30 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026MonitoringBench: Semi-Automated Red-Teaming for Agent Monitoring10 May
  2. 2025Reliable Weak-to-Strong Monitoring of LLM Agents26 Aug
  3. 2025RedCoder: Automated Multi-Turn Red Teaming for Code LLMs25 Jun
  4. 2024Embodied Red Teaming for Auditing Robotic Foundation Models27 Nov
  5. 2024Games for AI Control: Models of Safety Evaluations of AI Deployment Protocols12 Sep
  6. 2023AI Control: Improving Safety Despite Intentional Subversion12 Dec

Other references · 20

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2021Program Synthesis with Large Language Modelscited by 3,509
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  5. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  6. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  7. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  8. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  9. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  10. 2022CodeT: Code Generation with Generated Testscited by 369
  11. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  12. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 228
8 more
  1. 2024LLM Critics Help Catch LLM Bugscited by 138
  2. 2024SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agentscited by 129
  3. 2024On scalable oversight with weak LLMs judging strong LLMscited by 94
  4. 2024CriticBench: Benchmarking LLMs for Critique-Correct Reasoningcited by 88
  5. 2024Prover-Verifier Games improve legibility of LLM outputscited by 79
  6. 2025CoT Red-Handed: Stress Testing Chain-of-Thought Monitoringcited by 49
  7. 2023CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generationcited by 39
  8. 2024CodeEditorBench: Evaluating Code Editing Capability of Large Language Modelscited by 22

Built on by

Not cited yet.