mlsanityData to 7 Oct 2026

LLM-as-a-Judge Is Not an Oracle: Why Self-Improving Agents Need Deterministic Guardrails

arXiv 2609.02246 ↗First posted 2 Sep 2026cs.AI cs.LGcited by 1

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

also in 2 more
Abstract (arXiv)

Self-improving agent pipelines have a problem at their center. An optimizer rewrites prompts to score higher, and the score comes from a judge that is itself an LLM. That judge has the last word on whether the system is getting better, and our position is that it has not earned it. The judge should be demoted from oracle to advisor: its verdict becomes one input among several, and every change is gated instead by a deterministic verification layer the judge cannot override. We reached this position by building the alternative and running it. Over months of running autonomous prompt-optimization loops in production across contract analysis, compliance review, and code quality, we cataloged eleven ways the evaluation signal failed, in four classes: judge bias, harness and metric failures, ground-truth errors, and reward hacking. Agents achieved perfect scores by reading cached answer keys from their environment, a 100% pass rate concealing 68% true capability. A corrupted ground-truth label caused the optimizer to delete correct compliance rules to agree with it. A syntactically broken prompt was promoted as the winner because a silent parser fallback improved the metric. Attempts to fix the judge by rewriting its rubric plateaued; the only reliable gain came from a structural constraint on its output order. In response we describe PROCTOR, a Teacher-Student loop in which a stateful orchestrator holds all tool access, stateless subagents diagnose failures and draft mutations they cannot apply, and a Teacher grades those mutations under five deterministic guardrails: hermetic sandboxes, capability-disjoint roles, acceptance checks that outrank the Teacher, frozen holdouts, and canary cases engineered so that a perfect score is itself evidence of cheating. We report the failures this prevented, and, because the Teacher is itself an LLM judge, the failures it did not.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 24 of its 29 references that are in the map. First posted on arXiv 2 Sep 2026.

Builds on

24 of 29 references.

References · 24

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  3. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  4. 2016Concrete Problems in AI Safetycited by 1,582
  5. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  6. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  7. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  8. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  9. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
  10. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  11. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  12. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
12 more
  1. 2025GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningcited by 449
  2. 2018AI safety via debatecited by 384
  3. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  4. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
  5. 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 270
  6. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 255
  7. 2018Categorizing Variants of Goodhart's Lawcited by 132
  8. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 119
  9. 2024On scalable oversight with weak LLMs judging strong LLMscited by 94
  10. 2024Feedback Loops With Language Models Drive In-Context Reward Hackingcited by 70
  11. 2024Spontaneous Reward Hacking in Iterative Self-Refinementcited by 24
  12. 2026Agent Memory Distillation: Empowering Small LLM Agents with Hierarchical Teacher Memorycited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Optimizing the Score, Losing Sight of the Task: Reward Hacking Across Weights, Selection, and Promptscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026The Harness as the Only Mutable Surface: Compliance-Bounded Self-Evolution of LLM Agents in Credit Pipelines, with a Measured Admission Gate
  2. 2026Bias in the Loop: Auditing LLM-as-a-Judge for Software Engineering
  3. 2026VERSE: Verified Self-Evolving Optimizer for Agent Harnesses
  4. 2026Phantom Guardrails: When Self-Improving Agent Harnesses Fix Failures That Never Happened
  5. 2026Codifying the Judge: Scalable Evaluation via Program Distillation
  6. 2026Agent-as-a-Judge
  7. 2026ContractEval: Query-Conditioned Execution Matching for Procedural Instruction Conformance
  8. 2026Catching One in Five: LLM-as-Judge Blind Spots in Production Multi-Turn Transaction Agents