mlsanityData to 14 Oct 2026

Do Before You Judge: Self-Reference as a Pathway to Better LLM Evaluation

arXiv 2509.19880 ↗First posted 24 Sep 2025cs.CL cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

LLM-as-Judge frameworks are increasingly popular for AI evaluation, yet research findings on the relationship between models' generation and judgment abilities remain inconsistent. We investigate this relationship through systematic dataset- and instance-level analyses across 11 models and 21 diverse tasks. Despite both capabilities relying on the same underlying knowledge, our analyses reveal they are only weakly correlated, primarily due to LLMs' sensitivity to the responses being judged. To address this, we propose a self-reference-guided evaluation strategy that leverages a model's own answers as references. This approach significantly strengthens the correlation between generation and judgment abilities, offering a practical path to align these skills and providing a reliable proxy for model selection in evaluation tasks.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 27 of its 33 references that are in the map. First posted on arXiv 24 Sep 2025.

Builds on

27 of 33 references.

References · 27

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  6. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  7. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  8. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  9. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  10. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 550
  11. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  12. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
15 more
  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  2. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
  3. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 326
  4. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 323
  5. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
  6. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 316
  7. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMscited by 309
  8. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 269
  9. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 126
  10. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 72
  11. 2023MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluationcited by 57
  12. 2023OceanGPT: A Large Language Model for Ocean Science Taskscited by 43
  13. 2024ReIFE: Re-evaluating Instruction-Following Evaluationcited by 22
  14. 2024Debatrix: Multi-dimensional Debate Judge with Iterative Chronological Analysis Based on LLMcited by 21
  15. 2024Aligning Large Language Models by On-Policy Self-Judgmentcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026When Languages Disagree: Self-Evolving Multilingual LLM Judgescited by 2
  2. 2026Can LLMs Judge Better Than They Generate? Evaluating Task Asymmetry, Mechanistic Interpretability and Transferability for In-Context QAcited by 1