mlsanityData to 14 Oct 2026

Judging Is Not Enumerating: Silent Omissions in LLM-Authored Acceptable Sets

arXiv 2608.01000 ↗First posted 2 Aug 2026cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Language models are increasingly promoted from examinees to examiners: they write the test suites, answer keys, rubrics, and reward functions that define correctness for other systems. We measure the capability that role assumes and find it lacking under the protocol the role is usually deployed with, one-shot greedy authoring with no test-time reasoning. Across four reference constructions - two with complete finite truth, one with a hardened executable reference (HumanEval+/MBPP+), one with an explicitly incomplete lexical reference (WordNet) - models judge whether a candidate belongs far better than they author the set itself. On the incompleteness-proof algorithmic construction the gap is +0.34 to +0.29 F1 over a 24x parameter range and does not close; on executable code, models judging at F1 0.74-0.90 author suites admitting only 19-42% of oracle-correct solutions. A control locates the deficit: asked to emit the predicate rather than its extension, the same models reach F1 about 0.99. The failure is not missing knowledge or an inability to specify, but an inability to materialise the region a specification induces. The dominant error is omission, which resists audit: an over-inclusion is a token a reviewer can challenge, a missing member an absence whose discovery is the authoring problem itself. Models detect planted over-inclusions 6-7x more often than planted omissions, and a production deployment of 43,227 items fails omission-first at 10:1. Wired into RLVR, an authored key costs 1.9 points of accuracy against an exact oracle and 18.5 WordNet-relative (six paired seeds, p=0.031). Gating authored verifiers on a known-correct probe cuts false rejection from 58-92% to at most 5%, but keeps only 5-39% of suites. Repairing them instead, by rewriting each wrong expected value to what a reference execution returns, raises yield 3.3-10.6x across four author families.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 36 of its 62 references that are in the map. First posted on arXiv 2 Aug 2026.

Builds on

36 of 62 references.

References · 36

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2021Program Synthesis with Large Language Modelscited by 3,637
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  7. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  8. 2016Concrete Problems in AI Safetycited by 1,642
  9. 2024A Survey on LLM-as-a-Judgecited by 1,423
  10. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  11. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  12. 2024Self-Rewarding Language Modelscited by 681
24 more
  1. 2022PAL: Program-aided Language Modelscited by 652
  2. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  3. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
  4. 2023Efficient Guided Generation for Large Language Modelscited by 200
  5. 2024LLM Critics Help Catch LLM Bugscited by 147
  6. 2024On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Taskscited by 142
  7. 2025Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifierscited by 45
  8. 2025Answer Matching Outperforms Multiple Choice for Language Model Evaluationcited by 41
  9. 2026RubricBench: Aligning Model-Generated Rubrics with Human Standardscited by 28
  10. 2026LLMs Gaming Verifiers: RLVR can Lead to Reward Hackingcited by 28
  11. 2024SELF-[IN]CORRECT: LLMs Struggle with Discriminating Self-Generated Responsescited by 26
  12. 2024TOGLL: Correct and Strong Test Oracle Generation with LLMscited by 24
  13. 2025Shrinking the Generation-Verification Gap with Weak Verifierscited by 24
  14. 2025From Accuracy to Robustness: A Study of Rule- and Model-based Verifiers in Mathematical Reasoningcited by 22
  15. 2025TinyV: Reducing False Negatives in Verification Improves RL for LLM Reasoningcited by 21
  16. 2024Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Modelscited by 18
  17. 2025Rethinking Verification for LLM Code Generation: From Generation to Testingcited by 14
  18. 2024Do LLMs generate test oracles that capture the actual or the expected program behaviour?cited by 11
  19. 2025Scoring Verifiers: Evaluating Synthetic Verification for Code and Reasoningcited by 9
  20. 2025Hallucination to Consensus: Multi-Agent LLMs for End-to-End JUnit Test Generationcited by 9
  21. 2026Before the Model Learns the Bug:Fuzzing RLVR Verifierscited by 7
  22. 2024Large-scale cloze evaluation reveals that token prediction tasks are neither lexically nor semantically alignedcited by 5
  23. 2024Toward Automated Validation of Language Model Synthesized Test Cases using Semantic Entropycited by 4
  24. 2025Sequential Enumeration in Large Language Modelscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026Grading the Graders: Verification Autonomy Levels (L0-L5) for LLM Reasoning
  2. 2026Correct Chains, Wrong Answers: Dissociating Reasoning from Output in LLM Logic
  3. 2025Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs
  4. 2026Judges Follow the Reference: The Derivation Gap in Verifying Tool-Call Arguments
  5. 2026When Absence Is Evidence: Evaluating Completeness-Sensitive Negative Reasoning in Large Language Models
  6. 2026All Verdicts are Not Equal: Rethinking LLM Judge Reliability
  7. 2026Beyond Accuracy: Decomposing the Reasoning Efficiency of LLMs
  8. 2025Unspoken Hints: Accuracy Without Acknowledgement in LLM Reasoning