mlsanityData to 14 Oct 2026

Scaffold, Not Vocabulary? A Controlled, Two-Tier, Pre-Registered Study of a Popperian Code-Generation Skill

arXiv 2606.06454 ↗First posted 4 Jun 2026cs.SE cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Large language models increasingly write, review, and judge code, and a fast-growing practice equips them with prompt 'skills' that ask the model to reason like a scientist. A prominent example tells the model to act as a Popperian falsificationist, and such skills are reported to improve generated code. But these gains are almost always read off an LLM-as-a-judge, an instrument with documented positional, self-preference, and stylistic biases. We ask: if it appears to help, is the gain from the skill's Popperian content, or from the structure any scaffold imposes? We pre-register a two-tier ablation with three controls: a length-matched placebo, a labels-only scaffold that keeps the Popperian headers but strips the procedure, and an execution oracle (HumanEval+ unit tests), plus a vocabulary-halo sentinel and a same-model self-judge audit. On a frontier model (Claude Sonnet 4.6, N=163) all conditions sit near the benchmark ceiling and do not separate, so the pre-registered +5-point improvement is not supported (a ceiling-limited non-detection). On a small model (Qwen2.5-Coder-0.5B, N=164) structured arms lift best-of-eight correctness by 20-22 points, but the full skill shows no separable benefit over a labels-only scaffold (aggregate F@8=L@8 vs V@8=34.8%), and the placebo trails by only 2.4 points. A 0.5B self-judge applying the Popperian rubric does not beat random selection and concentrates 60% of its picks on one index. In the two settings tested, the skill's Popperian procedural content adds no separable execution-correctness benefit beyond a labels-only scaffold, so the gains track scaffold structure. We contribute a calibrated negative result and a reusable disambiguation protocol; the finding bounds an engineering claim about one prompt-skill family and is not an evaluation of Popperian methodology in general.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 28 of its 43 references that are in the map. First posted on arXiv 4 Jun 2026.

Builds on

28 of 43 references.

References · 28

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  3. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  4. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  5. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  6. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  7. 2023Teaching Large Language Models to Self-Debugcited by 988
  8. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  9. 2021The Values Encoded in Machine Learning Researchcited by 172
  10. 2023Can Large Language Models Really Improve by Self-critiquing Their Own Plans?cited by 108
  11. 2023Style Over Substance: Evaluation Biases for Large Language Modelscited by 70
  12. 2024Understanding the Dark Side of LLMs' Intrinsic Self-Correctioncited by 42
16 more
  1. 2025Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judgecited by 36
  2. 2024CodeJudge-Eval: Can Large Language Models be Good Judges in Code Understanding?cited by 30
  3. 2022REAL ML: Recognizing, Exploring, and Articulating Limitations of Machine Learning Researchcited by 20
  4. 2024Do Advanced Language Models Eliminate the Need for Prompt Engineering in Software Engineering?cited by 19
  5. 2025Position is Power: System Prompts as a Mechanism of Bias in Large Language Models (LLMs)cited by 19
  6. 2025Don't Judge Code by Its Cover: Exploring Biases in LLM Judges for Code Evaluationcited by 18
  7. 2024The Impact of Prompt Programming on Function-Level Code Generationcited by 13
  8. 2026Quantifying and Mitigating Self-Preference Bias of LLM Judgescited by 10
  9. 2024A Looming Replication Crisis in Evaluating Behavior in Language Models? Evidence and Solutionscited by 8
  10. 2023A Two-Sided Discussion of Preregistration of NLP Researchcited by 6
  11. 2024Analyzing Prompt Influence on Automated Method Generation: An Empirical Study with Copilotcited by 6
  12. 2026Bias in the Loop: Auditing LLM-as-a-Judge for Software Engineeringcited by 4
  13. 2026When Prompt Under-Specification Improves Code Correctness: An Exploratory Study of Prompt Wording and Structure Effects on LLM-Based Code Generationcited by 3
  14. 2026How LLMs Are Persuaded: A Few Attention Heads, Reroutedcited by 2
  15. 2026Quality Without Usefulness: LLM-Generated XAI Narratives as Trust Heuristics Rather Than Decision Aidscited by 2
  16. 2026Preregistration for Experiments with AI Agentscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Falsification, Not Exposure: An Internally Preregistered Placebo-Controlled Decomposition of Self-Repair Feedback in Frozen Small Code Modelscited by 4
  2. 2026Selection Without Signal, Recovery Through Expression: A Measurement Study of Post-Hoc Falsification Operators for Frozen Small Code Modelscited by 2
  3. 2026Form, Not Content? A Preregistered, Placebo-Controlled Evaluation of Learned Error-Conditioned Self-Repair Through Prompts and Weights in Frozen Small Code Modelscited by 2