mlsanityData to 14 Oct 2026

Selection Without Signal, Recovery Through Expression: A Measurement Study of Post-Hoc Falsification Operators for Frozen Small Code Models

arXiv 2606.16999 ↗First posted 15 Jun 2026cs.SE cs.CL cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Frozen small code models (<=1.5B parameters, run locally without fine-tuning) suit offline and privacy-constrained use, but often emit plausible-but-wrong programs. A natural remedy is a post-hoc operator that selects, verifies, repairs, or re-processes the model's samples without retraining; in principled form it is Popperian: attack each candidate with a severe test, keep what survives. We measure whether such operators help. Under one deterministic execution oracle and a leakage-free, matched-compute protocol, 26 semantic post-hoc operators (selection, verification, repair, elimination, portfolios, sound vetoes, generation conditioning) are evaluated against Best-of-N (BoN); on the cells and benchmarks tested, none improves held-out accuracy over BoN. The negative is mechanistic: a coverage wall (systematic hard-task failures deeper sampling does not rescue), a capability scissors (a competent generator leaves almost no discriminable error among visible-test passers), and a near-empty consensus trap (the visible-pass-but-hidden-wrong majority a leakage-free selector needs rarely co-occurs with a correct alternative). A distribution-free do-no-harm bound cannot certify a harm rate <=alpha at zero observed harm unless n>=45. Two operators help on a different axis, outside the semantic output space. An expression-layer recovery (M1), the only accuracy gain here, recovers correct programs the standard extractor discards (robust extraction and public-test signature alignment); it does no harm (b10=0), is leakage-free, and lifts DeepSeek-Coder-1.3B by +12 tasks on HumanEval+ (p=2.4e-4). An adaptive consensus early-stop (ACE) is a calibrated compute-saving control (~19% saving, zero harm). M1 and the selection negative replicate on HumanEval+ and MBPP+ across three model cells. The lesson: fix the harness and measure coverage before blaming semantic post-hoc reasoning.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 33 of its 46 references that are in the map. First posted on arXiv 15 Jun 2026.

Builds on

33 of 46 references.

References · 33

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  4. 2023Let's Verify Step by Stepcited by 3,789
  5. 2021Program Synthesis with Large Language Modelscited by 3,637
  6. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  7. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  8. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  9. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  10. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  11. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  12. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
21 more
  1. 2023Teaching Large Language Models to Self-Debugcited by 988
  2. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  3. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  4. 2022CodeT: Code Generation with Generated Testscited by 384
  5. 2021Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Controlcited by 271
  6. 2022Natural Language to Code Translation with Executioncited by 124
  7. 2022Coder Reviewer Reranking for Code Generationcited by 84
  8. 2024Planning In Natural Language Improves LLM Search For Code Generationcited by 83
  9. 2024Learning How Hard to Think: Input-Adaptive Allocation of LM Computationcited by 70
  10. 2025Correlated Errors in Large Language Modelscited by 45
  11. 2025Automated Hypothesis Validation with Agentic Sequential Falsificationscited by 44
  12. 2024Understanding the Dark Side of LLMs' Intrinsic Self-Correctioncited by 42
  13. 2024On Leakage of Code Generation Evaluation Datasetscited by 32
  14. 2025Revisit Self-Debugging with Self-Generated Tests for Code Generationcited by 22
  15. 2026CoVerRL: Breaking the Consensus Trap in Label-Free Reasoning via Generator-Verifier Co-Evolutioncited by 12
  16. 2026SEVerA: Verified Synthesis of Self-Evolving Agentscited by 8
  17. 2025Can Language Models Falsify? Evaluating Algorithmic Reasoning with Counterexample Creationcited by 7
  18. 2025Uncovering Systematic Failures of LLMs in Verifying Code Against Natural Language Specificationscited by 5
  19. 2026Scaffold, Not Vocabulary? A Controlled, Two-Tier, Pre-Registered Study of a Popperian Code-Generation Skillcited by 3
  20. 2026VIBEPASS: Can Vibe Coders Really Pass the Vibe Check?cited by 2
  21. 2026MUCOCO: Automated Consistency Testing of Code LLMscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Falsification, Not Exposure: An Internally Preregistered Placebo-Controlled Decomposition of Self-Repair Feedback in Frozen Small Code Modelscited by 4
  2. 2026Form, Not Content? A Preregistered, Placebo-Controlled Evaluation of Learned Error-Conditioned Self-Repair Through Prompts and Weights in Frozen Small Code Modelscited by 2