# Verification Trap: Understanding Test-Time Selection Failures under False Premises in Code Generation

arXiv: https://arxiv.org/abs/2610.05170 · HTML: https://mlsanity.com/paper/2610.05170/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 34 references that are in the map.
- First posted on arXiv 4 Oct 2026.

## Placed in

- LLM reasoning

## Builds on (most cited first, 25 of 25)

- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Large Language Monkeys: Scaling Inference Compute with Repeated Sampling](https://mlsanity.com/paper/2407.21787/) (2024)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models](https://mlsanity.com/paper/2310.04406/) (2023)
- [CodeT: Code Generation with Generated Tests](https://mlsanity.com/paper/2207.10397/) (2022)
- [The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models](https://mlsanity.com/paper/2201.03544/) (2022)
- [LEVER: Learning to Verify Language-to-Code Generation with Execution](https://mlsanity.com/paper/2302.08468/) (2023)
- [Making Large Language Models Better Reasoners with Step-Aware Verifier](https://mlsanity.com/paper/2206.02336/) (2022)
- [Is Self-Repair a Silver Bullet for Code Generation?](https://mlsanity.com/paper/2306.09896/) (2023)
- [Planning with Large Language Models for Code Generation](https://mlsanity.com/paper/2303.05510/) (2023)
- [Natural Language to Code Translation with Execution](https://mlsanity.com/paper/2204.11454/) (2022)
- [S*: Test Time Scaling for Code Generation](https://mlsanity.com/paper/2502.14382/) (2025)
- [Coder Reviewer Reranking for Code Generation](https://mlsanity.com/paper/2211.16490/) (2022)
- [CREPE: Open-Domain Question Answering with False Presuppositions](https://mlsanity.com/paper/2211.17257/) (2022)
- [Demonstrating specification gaming in reasoning models](https://mlsanity.com/paper/2502.13295/) (2025)
- [Won't Get Fooled Again: Answering Questions with False Premises](https://mlsanity.com/paper/2307.02394/) (2023)
- [Don't Let It Hallucinate: Premise Verification via Retrieval-Augmented Logical Reasoning](https://mlsanity.com/paper/2504.06438/) (2025)
- [SWE-Replay: Efficient Test-Time Scaling for Software Engineering Agents](https://mlsanity.com/paper/2601.22129/) (2026)
- [Scaling Agentic Verifier for Competitive Coding](https://mlsanity.com/paper/2602.04254/) (2026)
- [DecoPrompt : Decoding Prompts Reduces Hallucinations when Large Language Models Meet False Premises](https://mlsanity.com/paper/2411.07457/) (2024)
- [DAJ: Data-Reweighted LLM Judge for Test-Time Scaling in Code Generation](https://mlsanity.com/paper/2601.22230/) (2026)
- [CVeDRL: An Efficient Code Verifier via Difficulty-aware Reinforcement Learning](https://mlsanity.com/paper/2601.22803/) (2026)

## Abstract (arXiv)

Test-time compute has become a central way to improve code generation: systems sample multiple candidate programs and use verifier-visible evidence to select the final output. This paradigm implicitly assumes that the verifier provides a corrective signal independent from the generator. We challenge this assumption under misleading task premises. When the generator and verifier share a false premise, they become coupled through a mistaken belief: the generator produces premise-consistent shortcuts, while the verifier supplies evidence that fails to expose them. Consequently, the selector may choose a hidden-test-wrong candidate even when a hidden-test-correct program exists in the pool. We call this failure mode Verification Trap. Across three code-generation benchmarks and five code models, false premises consistently degrade first-sample correctness, reduce selector-chosen correctness after 64-sample test-time selection, and amplify recoverable mis-selection. Mechanistically, verifier-written tests inherit the premise-level blind spot, reshaping verifier-visible candidate space away from hidden-test correctness. These traces make Verification Trap predictable before hidden execution: a lightweight gold-free predictor using verifier-visible features reaches 0.846 AUROC. Our results identify decoupled evidence as a key mitigation axis: coupled scaling provides limited recovery, whereas premise-agnostic robustness auditors recover substantial oracle headroom.