mlsanityData to 14 Oct 2026

Cross-Context Verification: Hierarchical Detection of Benchmark Contamination through Session-Isolated Analysis

arXiv 2603.21454 ↗First posted 23 Mar 2026cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

LLM coding benchmarks face a credibility crisis: widespread solution leakage and test quality issues undermine SWE-bench Verified, while existing detection methods--paraphrase consistency, n-gram overlap, perplexity analysis--never directly observe whether a model reasons or recalls. Meanwhile, simply repeating verification degrades accuracy: multi-turn review generates false positives faster than it discovers true errors, suggesting that structural approaches are needed. We introduce Cross-Context Verification (CCV), a black-box method that solves the same benchmark problem in N independent sessions and measures solution diversity, combined with the Hierarchical Cross-Context Architecture (HCCA), a multi-agent analysis framework that prevents confirmation bias through intentional information restriction across specialized analytical roles. On 9 SWE-bench Verified problems (45 trials, Claude Opus 4.6, temperature 0), CCV achieves perfect separation between contaminated and genuine reasoning (Mann-Whitney U=0, p approx 0.012, r = 1.0). Key findings: (1) contamination is binary--models either recall perfectly or not at all; (2) reasoning absence is a perfect discriminator; (3) 33% of prior contamination labels are false positives; (4) HCCA's independent analysis structure discovers contamination-flaw composite cases that single-analyst approaches miss. A pilot experiment extending HCCA to multi-stage verification (Worker to Verifier to Director) yields a negative result--100% sycophantic confirmation--providing further evidence that information restriction, not structural complexity, is the key mechanism. We release all code and data.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 10 of its 14 references that are in the map. First posted on arXiv 23 Mar 2026.

Builds on

10 of 14 references.

References · 10

  1. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  3. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  4. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  5. 2024SWE-Bench+: Enhanced Coding Benchmark for LLMscited by 99
  6. 2024Non-Determinism of "Deterministic" LLM Settingscited by 92
  7. 2025The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reasoncited by 49
  8. 2023Estimating Contamination via Perplexity: Quantifying Memorisation in Language Model Evaluationcited by 48
  9. 2026Cross-Context Review: Improving LLM Output Quality by Separating Production and Review Sessionscited by 5
  10. 2026More Rounds, More Noise: Why Multi-Turn Review Fails to Improve Cross-Context Verificationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.