Refute-or-Promote: An Adversarial Stage-Gated Multi-Agent Review Methodology for High-Precision LLM-Assisted Defect Discovery
On the map: LLM agents and tool use → Web, search, and research agents
Abstract (arXiv)
LLM-assisted defect discovery has a precision crisis: plausible-but-wrong reports overwhelm maintainers and degrade credibility for real findings. We present Refute-or-Promote, an inference-time reliability pattern combining Stratified Context Hunting (SCH) for candidate generation, adversarial kill mandates, context asymmetry, and a Cross-Model Critic (CMC). Adversarial agents attempt to disprove candidates at each promotion gate; cold-start reviewers are intended to reduce anchoring cascades; cross-family review can catch correlated blind spots that same-family review misses. Over a 31-day campaign across 7 targets (security libraries, the ISO C++ standard, major compilers), the pipeline killed roughly 79% of 171 candidates before advancing to disclosure (retrospective aggregate); on a consolidated-protocol subset (lcms2, wolfSSL; n=30), the prospective kill rate was 83%. Outcomes: 4 CVEs (3 public, 1 embargoed); LWG 4549 accepted to the C++ working paper; 5 merged C++ editorial PRs; 3 compiler conformance bugs; 8 merged security-related fixes without CVE; an RFC 9000 errata filed under committee review; and 1+ FIPS 140-3 normative compliance issues under coordinated disclosure -- all evaluated by external acceptance, not benchmarks. The most instructive failure: ten dedicated reviewers unanimously endorsed a non-existent Bleichenbacher padding oracle in OpenSSL's CMS module; it was killed only by a single empirical test, motivating the mandatory empirical gate. No vulnerability was discovered autonomously; the contribution is external structure that filters LLM agents' persistent false positives. As a preliminary transfer test beyond defect discovery, a simplified cross-family critique variant also solved five previously unsolved SymPy instances on SWE-bench Verified and one SWE-rebench hard task.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 28 of its 51 references that are in the map. First posted on arXiv 21 Apr 2026.
Builds on
28 of 51 references.
References · 28
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
- 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
- 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
- 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 420
- 2018AI safety via debatecited by 384
- 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 255
- 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 222
- 2024IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilitiescited by 100
- 2025Talk Isn't Always Cheap: Understanding Failure Modes in Multi-Agent Debatecited by 63
- 2025Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifierscited by 58
- 2025Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?cited by 52
16 more
- 2025SWE-Debate: Competitive Multi-Agent Debate for Software Issue Resolutioncited by 52
- 2025RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditingcited by 47
- 2025Correlated Errors in Large Language Modelscited by 42
- 2025Let the Trial Begin: A Mock-Court Approach to Vulnerability Detection using LLM-Based Agentscited by 20
- 2025Beyond Consensus: Mitigating the Agreeableness Bias in LLM Judge Evaluationscited by 16
- 2024Debate, Deliberate, Decide (D3): A Cost-Aware Adversarial Framework for Reliable and Interpretable LLM Evaluationcited by 14
- 2025VulAgent: Hypothesis-Validation based Multi-Agent Vulnerability Detectioncited by 8
- 2026Consensus is Not Verification: Why Crowd Wisdom Strategies Fail for LLM Truthfulnesscited by 8
- 2025InfCode: Adversarial Iterative Refinement of Tests and Patches for Reliable Software Issue Resolutioncited by 7
- 2025MAVUL: Multi-Agent Vulnerability Detection via Contextual Reasoning and Interactive Refinementcited by 6
- 2026Reducing False Positives in Static Bug Detection with LLMs: An Empirical Study in Industrycited by 5
- 2026Sifting the Noise: A Comparative Study of LLM Agents in Vulnerability False Positive Filteringcited by 5
- 2026Cross-Context Review: Improving LLM Output Quality by Separating Production and Review Sessionscited by 5
- 2026More Rounds, More Noise: Why Multi-Turn Review Fails to Improve Cross-Context Verificationcited by 4
- 2026A Content-Based Framework for Cybersecurity Refusal Decisions in Large Language Modelscited by 1
- 2026AEGIS: From Clues to Verdicts -- Graph-Guided Deep Vulnerability Reasoning via Dialectics and Meta-Auditingcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Test vs Mutant: Adversarial LLM Agents for Robust Unit Test Generation
- 2026Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents
- 2026How Much Can We Trust LLM Search Agents? Measuring Endorsement Vulnerability to Web Content Manipulation
- 2026Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
- 2026DECEIVE-AFC: Adversarial Claim Attacks against Search-Enabled LLM-based Fact-Checking Systems
- 2026Adversarial Review: Structured Disagreement for Grounded Agentic Code Review
- 2026Detection Without Correction: A Two-Parameter Decomposition of Multi-Stage LLM Pipelines
- 2026Reviewer Capability Governs Rejection Targeting, Not Repair Skill: Evidence from LLM Execute-Review-Revise Pipelines