# An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4

arXiv: https://arxiv.org/abs/2403.02839 · HTML: https://mlsanity.com/paper/2403.02839/

## Key facts

- Cited by 94 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 37 references that are in the map.
- First posted on arXiv 5 Mar 2024.

## Builds on (most cited first, 22 of 22)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate](https://mlsanity.com/paper/2308.07201/) (2023)
- [A Survey on In-context Learning](https://mlsanity.com/paper/2301.00234/) (2022)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [Prometheus: Inducing Fine-grained Evaluation Capability in Language Models](https://mlsanity.com/paper/2310.08491/) (2023)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [JudgeLM: Fine-tuned Large Language Models are Scalable Judges](https://mlsanity.com/paper/2310.17631/) (2023)
- [PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization](https://mlsanity.com/paper/2306.05087/) (2023)
- [Evaluating Large Language Models at Evaluating Instruction Following](https://mlsanity.com/paper/2310.07641/) (2023)
- [ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversation](https://mlsanity.com/paper/2310.17389/) (2023)
- [Probing Neural Network Comprehension of Natural Language Arguments](https://mlsanity.com/paper/1907.07355/) (2019)
- [Branch-Solve-Merge Improves Large Language Model Evaluation and Generation](https://mlsanity.com/paper/2310.15123/) (2023)
- [CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generation](https://mlsanity.com/paper/2311.18702/) (2023)

## Built on by (most cited first, 30 of 94)

- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge](https://mlsanity.com/paper/2410.02736/) (2024)
- [Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models](https://mlsanity.com/paper/2404.18796/) (2024)
- [Large Language Models are Inconsistent and Biased Evaluators](https://mlsanity.com/paper/2405.01724/) (2024)
- [Benchmark Data Contamination of Large Language Models: A Survey](https://mlsanity.com/paper/2406.04244/) (2024)
- [AutoEval Done Right: Using Synthetic Data for Model Evaluation](https://mlsanity.com/paper/2403.07008/) (2024)
- [Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks](https://mlsanity.com/paper/2510.27106/) (2025)
- [Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language models](https://mlsanity.com/paper/2405.02287/) (2024)
- [How to Correctly Report LLM-as-a-Judge Evaluations](https://mlsanity.com/paper/2511.21140/) (2025)
- [Noisy but Valid: Robust Statistical Evaluation of LLMs with Imperfect Judges](https://mlsanity.com/paper/2601.20913/) (2026)
- [Think-J: Learning to Think for Generative LLM-as-a-Judge](https://mlsanity.com/paper/2505.14268/) (2025)
- [TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them](https://mlsanity.com/paper/2509.21117/) (2025)
- [Visual-RAG: Benchmarking Text-to-Image Retrieval Augmented Generation for Visual Knowledge Intensive Queries](https://mlsanity.com/paper/2502.16636/) (2025)
- [Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models](https://mlsanity.com/paper/2505.16265/) (2025)
- [Is Relevance Propagated from Retriever to Generator in RAG?](https://mlsanity.com/paper/2502.15025/) (2025)
- [MedGUIDE: Benchmarking Clinical Decision-Making in Large Language Models](https://mlsanity.com/paper/2505.11613/) (2025)
- [Automatically Benchmarking LLM Code Agents through Agent-Driven Annotation and Evaluation](https://mlsanity.com/paper/2510.24358/) (2025)
- [Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimization](https://mlsanity.com/paper/2603.08091/) (2026)
- [When LLM Judge Scores Look Good but Best-of-N Decisions Fail](https://mlsanity.com/paper/2603.12520/) (2026)
- [Enhancing LLM-based Search Agents via Contribution Weighted Group Relative Policy Optimization](https://mlsanity.com/paper/2604.14267/) (2026)
- [Reframing LLM Agent Security as an Agent-Human Interaction Problem](https://mlsanity.com/paper/2605.24309/) (2026)
- [Can LLMs Automate Fact-Checking Article Writing?](https://mlsanity.com/paper/2503.17684/) (2025)
- [Bridging Human and LLM Judgments: Understanding and Narrowing the Gap](https://mlsanity.com/paper/2508.12792/) (2025)
- [PAIR-SAFE: A Paired-Agent Approach for Runtime Auditing and Refining AI-Mediated Mental Health Support](https://mlsanity.com/paper/2601.12754/) (2026)
- [VCSearch: Bridging the Gap Between Well-Defined and Ill-Defined Problems in Mathematical Reasoning](https://mlsanity.com/paper/2406.05055/) (2024)
- [Unveiling and Mitigating Bias in Large Language Model Recommendations: A Path to Fairness](https://mlsanity.com/paper/2409.10825/) (2024)
- [Quantitative LLM Judges](https://mlsanity.com/paper/2506.02945/) (2025)
- [TURA: Tool-Augmented Unified Retrieval Agent for AI Search](https://mlsanity.com/paper/2508.04604/) (2025)
- [Mind the Generation Process: Fine-Grained Confidence Estimation During LLM Generation](https://mlsanity.com/paper/2508.12040/) (2025)
- [Can AI Truly Represent Your Voice in Deliberations? A Comprehensive Study of Large-Scale Opinion Aggregation with LLMs](https://mlsanity.com/paper/2510.05154/) (2025)

## Abstract (arXiv)

Recently, there has been a growing trend of utilizing Large Language Model (LLM) to evaluate the quality of other LLMs. Many studies have fine-tuned judge models based on open-source LLMs for evaluation. While the fine-tuned judge models are claimed to achieve comparable evaluation capability with GPT-4, in this work, we conduct an empirical study of LLM-as-a-Judge. Our findings indicate that although the fine-tuned judge models achieve high performance on in-domain test sets, even surpassing GPT-4, they underperform GPT-4 across several dimensions, including generalizability, fairness and adaptability. We also reveal that the fine-tuned judge model inherently operates as a task-specific classifier, consequently imposing the limitations.