# Competence, Not Accuracy: A Diagnostic for Reference-Free Judge Gates in Skill Optimization

arXiv: https://arxiv.org/abs/2608.18719 · HTML: https://mlsanity.com/paper/2608.18719/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 19 references that are in the map.
- First posted on arXiv 19 Aug 2026.

## Builds on (most cited first, 14 of 14)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning](https://mlsanity.com/paper/2507.19457/) (2025)
- [SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning](https://mlsanity.com/paper/2602.08234/) (2026)
- [Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skills](https://mlsanity.com/paper/2603.25158/) (2026)
- [Time Travel in LLMs: Tracing Data Contamination in Large Language Models](https://mlsanity.com/paper/2308.08493/) (2023)
- [CoEvoSkills: Self-Evolving Agent Skills via Co-Evolutionary Verification](https://mlsanity.com/paper/2604.01687/) (2026)
- [EvoSkill: Automated Skill Discovery for Multi-Agent Systems](https://mlsanity.com/paper/2603.02766/) (2026)
- [SkillClaw: Let Skills Evolve Collectively with Agentic Evolver](https://mlsanity.com/paper/2604.08377/) (2026)
- [SkillX: Automatically Constructing Skill Knowledge Bases for Agents](https://mlsanity.com/paper/2604.04804/) (2026)
- [SealQA: Raising the Bar for Reasoning in Search-Augmented Language Models](https://mlsanity.com/paper/2506.01062/) (2025)
- [Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems](https://mlsanity.com/paper/2502.19328/) (2025)
- [Dynamic Dual-Granularity Skill Bank for Agentic RL](https://mlsanity.com/paper/2603.28716/) (2026)
- [SkillSmith: Co-Evolving Skills and Tools for Self-Improving Agent Systems](https://mlsanity.com/paper/2606.01314/) (2026)

## Abstract (arXiv)

Text-space skill optimization adapts a frozen agent by evolving a natural-language skill document, accepting each candidate through a validation gate. Existing gates rely on verifiable rewards, confining these methods to tasks with an automatic verifier. Replacing the verifier with an LLM-judge gate would lift that restriction, but whether such a gate carries usable signal is untested. We ask a prior question: can we tell, before placing a judge in the loop, whether its scores separate correct from incorrect answers at all? We formalize a reference-free judge as a latent solver -- its verdict rests on agreement with whatever it would itself conclude, so its capacity to evaluate is bounded by its capacity to solve. The model yields a closed-form bound on discriminability (ROC-AUC) in the judge's competence c and answer-space size k, a necessary condition c > 1/k, and the result that the marginal AUC is confounded by item difficulty while a within-question estimator is not. A non-intervening probe records judge scores on genuine optimization runs without altering any decision. We find discriminability at chance where competence sits near the floor and usable above it; that a judge's benchmark accuracy overstates the competence that matters; and, in a closed-loop study, that the screen predicts which kind of gating error occurs. The result is a cheap pre-deployment diagnostic for judge gates.