mlsanityData to 14 Oct 2026

GRP: Goal-Reversed Prompting for Zero-Shot Evaluation with LLMs

arXiv 2503.06139 ↗First posted 8 Mar 2025cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Pairwise LLM-as-a-judge evaluation asks the judge to identify the better of two candidate answers. We study a one-line modification that asks for the worse answer instead and recovers the preference by elimination, a procedure we call Goal-Reversed Prompting (GRP). GRP introduces no extra inference rounds, composes with any prompt template (direct, chain-of-thought, or Arena-Hard SOP), and leaves the rest of the evaluation pipeline untouched. Two observations motivate the reversal. Reverse reasoning is a recurring strategy in human problem solving, and modern instruction-tuned judges exhibit a positive-leaning bias that asking for the worse answer can counteract. On JudgeBench under a strict consistency protocol that counts a judgment as correct only when both response orderings agree with the gold preference, GRP improves all three closed-source judges we test across both response-pair sources. With GPT-4o-generated pairs, the Arena-Hard SOP baseline improves from 61.71% to 66.23% for GPT-4o (+4.52) and from 60.00% to 66.00% for Claude-3.5-Sonnet (+6.00), with the largest absolute gains on Reasoning and Mathematics. The lift persists when response pairs come from Claude-3.5-Sonnet and when the SOP scaffolding is stripped to a minimal direct-prompting template, suggesting that goal reversal acts on the underlying judging behavior rather than on a particular rubric. Stronger judges benefit more than weaker ones, suggesting that goal reversal exposes additional reasoning capacity rather than compensating for its absence.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 11 of its 15 references that are in the map. First posted on arXiv 8 Mar 2025.

Builds on

11 of 15 references.

References · 11

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  3. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
  4. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 323
  5. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
  6. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 150
  7. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 90
  8. 2023Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Modelscited by 87
  9. 2024Reverse Thinking Makes LLMs Stronger Reasonerscited by 29
  10. 2023Are Large Language Models Reliable Judges? A Study on the Factuality Evaluation Capabilities of LLMscited by 24
  11. 2024Can Many-Shot In-Context Learning Help LLMs as Evaluators? A Preliminary Empirical Studycited by 15

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026RTLC -- Research, Teach-to-Learn, Critique: A three-stage prompting paradigm inspired by the Feynman Learning Technique that lifts LLM-as-judge accuracy on JudgeBench with no fine-tuning
  2. 2025FOR-Prompting: From Objection to Revision via an Asymmetric Prompting Protocol
  3. 2026Will the Judge Flip? Predicting Position-Sensitive LLM Judgments from Residual Stream Activations
  4. 2026From Judgment Quality to Downstream Utility: Rethinking LLM-as-a-Judge for Open-Ended Tasks
  5. 2026The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation
  6. 2025You Don't Need Prompt Engineering Anymore: The Prompting Inversion
  7. 2026Measuring Evaluation-Context Divergence in Open-Weight LLMs: A Paired-Prompt Protocol with Pilot Evidence of Alignment-Pipeline-Specific Heterogeneity
  8. 2026ReverseMath: Answer Inversion for Scalable and Verifiable Mathematical Problem Generation