# Graders should cheat: privileged information enables expert-level automated evaluations

arXiv: https://arxiv.org/abs/2502.10961 · HTML: https://mlsanity.com/paper/2502.10961/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 14 references that are in the map.
- First posted on arXiv 16 Feb 2025.

## Builds on (most cited first, 8 of 8)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [Evaluating Large Language Models at Evaluating Instruction Following](https://mlsanity.com/paper/2310.07641/) (2023)
- [Generative Judge for Evaluating Alignment](https://mlsanity.com/paper/2310.05470/) (2023)
- [Exploring the Reliability of Large Language Models as Customized Evaluators for Diverse NLP Tasks](https://mlsanity.com/paper/2310.19740/) (2023)
- [RevisEval: Improving LLM-as-a-Judge via Response-Adapted References](https://mlsanity.com/paper/2410.05193/) (2024)
- [How Reliable Are Automatic Evaluation Methods for Instruction-Tuned LLMs?](https://mlsanity.com/paper/2402.10770/) (2024)

## Built on by (most cited first, 5 of 5)

- [ECLeKTic: a Novel Challenge Set for Evaluation of Cross-Lingual Knowledge Transfer](https://mlsanity.com/paper/2502.21228/) (2025)
- [S2J: Bridging the Gap Between Solving and Judging Ability in Generative Reward Models](https://mlsanity.com/paper/2509.22099/) (2025)
- [ORCA-bench: How Ready Are Language Model Agents for Oncall?](https://mlsanity.com/paper/2607.28545/) (2026)
- [When Vision-Language Models Judge Without Seeing: Exposing Informativeness Bias](https://mlsanity.com/paper/2604.17768/) (2026)
- [How to Trick Your AI TA: A Systematic Study of Academic Jailbreaking in LLM Code Evaluation](https://mlsanity.com/paper/2512.10415/) (2025)

## Abstract (arXiv)

Auto-evaluating language models (LMs), i.e., using a grader LM to evaluate the candidate LM, is an appealing way to accelerate the evaluation process and the cost associated with it. But this presents a paradox: how can we trust the grader LM, which is presumably weaker than the candidate LM, to assess problems that are beyond the frontier of the capabilities of either model or both? For instance, today's LMs struggle on graduate-level physics and Olympiad-level math, making them unreliable graders in these domains. We show that providing privileged information -- such as ground-truth solutions or problem-specific guidelines -- improves automated evaluations on such frontier problems. This approach offers two key advantages. First, it expands the range of problems where LMs graders apply. Specifically, weaker models can now rate the predictions of stronger models. Second, privileged information can be used to devise easier variations of challenging problems which improves the separability of different LMs on tasks where their performance is generally low. With this approach, general-purpose LM graders match the state of the art performance on RewardBench, surpassing almost all the specially-tuned models. LM graders also outperform individual human raters on Vibe-Eval, and approach human expert graders on Olympiad-level math problems.