mlsanityData to 14 Oct 2026

JudgeLM: Fine-tuned Large Language Models are Scalable Judges

arXiv 2310.17631 ↗First posted 26 Oct 2023cs.CL cs.AIcited by 367

Not on the map. Why?

Cited by 40 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Evaluating Large Language Models (LLMs) in open-ended scenarios is challenging because existing benchmarks and metrics can not measure them comprehensively. To address this problem, we propose to fine-tune LLMs as scalable judges (JudgeLM) to evaluate LLMs efficiently and effectively in open-ended benchmarks. We first propose a comprehensive, large-scale, high-quality dataset containing task seeds, LLMs-generated answers, and GPT-4-generated judgments for fine-tuning high-performance judges, as well as a new benchmark for evaluating the judges. We train JudgeLM at different scales from 7B, 13B, to 33B parameters, and conduct a systematic analysis of its capabilities and behaviors. We then analyze the key biases in fine-tuning LLM as a judge and consider them as position bias, knowledge bias, and format bias. To address these issues, JudgeLM introduces a bag of techniques including swap augmentation, reference support, and reference drop, which clearly enhance the judge's performance. JudgeLM obtains the state-of-the-art judge performance on both the existing PandaLM benchmark and our proposed new benchmark. Our JudgeLM is efficient and the JudgeLM-7B only needs 3 minutes to judge 5K samples with 8 A100 GPUs. JudgeLM obtains high agreement with the teacher judge, achieving an agreement exceeding 90% that even surpasses human-to-human agreement. JudgeLM also demonstrates extended capabilities in being judges of the single answer, multimodal models, multiple answers, multi-turn chat, etc. Code is available at https://github.com/baaivision/JudgeLM.

Key facts. Cited by 367 papers in the map (data through 14 Oct 2026). Builds on 34 of its 56 references that are in the map. First posted on arXiv 26 Oct 2023.

Builds on

34 of 56 references.

References · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2023Visual Instruction Tuningcited by 7,869
  10. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,288
22 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  2. 2023Qwen Technical Reportcited by 3,344
  3. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  5. 2020Knowledge Distillation: A Surveycited by 1,603
  6. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  7. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  8. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  9. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  10. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  11. 2023Instruction Tuning with GPT-4cited by 686
  12. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  13. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  14. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  15. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 316
  16. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 236
  17. 2023MathPrompter: Mathematical Reasoning using Large Language Modelscited by 236
  18. 2023Generative Judge for Evaluating Alignmentcited by 169
  19. 2023Emu: Generative Pretraining in Multimodalitycited by 157
  20. 2020Look at the First Sentence: Position Bias in Question Answeringcited by 84
  21. 2023TIGERScore: Towards Building Explainable Metric for All Text Generation Taskscited by 70
  22. 2023INSTRUCTSCORE: Explainable Text Generation Evaluation with Finegrained Feedbackcited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 70; 2025: 144; 2026: 153.

367 papers.

Sort:

By area:

  1. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  2. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  3. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  4. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  5. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 372
  6. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 326
  7. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
  8. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 269
  9. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 205
  10. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 147
  11. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judgecited by 143
  12. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 126
38 more
  1. 2024Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluationcited by 125
  2. 2025Process Reward Models That Thinkcited by 118
  3. 2024Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreementcited by 105
  4. 2025TRAIL: Trace Reasoning and Agentic Issue Localizationcited by 104
  5. 2024Higher Layers Need More LoRA Expertscited by 102
  6. 2024An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4cited by 94
  7. 2024CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklistscited by 94
  8. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
  9. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 93
  10. 2025OmniSQL: Synthesizing High-quality Text-to-SQL Data at Scalecited by 91
  11. 2025Fin-R1: A Large Language Model for Financial Reasoning through Reinforcement Learningcited by 83
  12. 2024RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculturecited by 81
  13. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 78
  14. 2025The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMscited by 74
  15. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 73
  16. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 72
  17. 2024Eliminating Position Bias of Language Models: A Mechanistic Approachcited by 65
  18. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Samplingcited by 60
  19. 2024Lynx: An Open Source Hallucination Evaluation Modelcited by 57
  20. 2024DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processingcited by 52
  21. 2024LLaMoCo: Instruction Tuning of Large Language Models for Optimization Code Generationcited by 45
  22. 2024HelloBench: Evaluating Long Text Generation Capabilities of Large Language Modelscited by 44
  23. 2024CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolutioncited by 42
  24. 2025Answer Matching Outperforms Multiple Choice for Language Model Evaluationcited by 41
  25. 2025CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Rewardcited by 40
  26. 2025RocketEval: Efficient Automated LLM Evaluation via Grading Checklistcited by 39
  27. 2025Do LLM Evaluators Prefer Themselves for a Reason?cited by 38
  28. 2025Improving LLM-as-a-Judge Inference with the Judgment Distributioncited by 37
  29. 2025LLMs Cannot Reliably Judge (Yet?): A Comprehensive Assessment on the Robustness of LLM-as-a-Judgecited by 37
  30. 2025Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judgecited by 36
  31. 2024CalibraEval: Calibrating Prediction Distribution to Mitigate Selection Bias in LLMs-as-Judgescited by 34
  32. 2024Decoding Biases: Automated Methods and LLM Judges for Gender Bias Detection in Language Modelscited by 33
  33. 2025CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Taskscited by 33
  34. 2024Finding Blind Spots in Evaluator LLMs with Interpretable Checklistscited by 32
  35. 2025Beyond the Surface: Measuring Self-Preference in LLM Judgmentscited by 31
  36. 2025How Reliable is Multilingual LLM-as-a-Judge?cited by 30
  37. 2025Multi-Agent-as-Judge: Aligning LLM-Agent-Based Automated Evaluation with Multi-Dimensional Human Evaluationcited by 30
  38. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Modelscited by 29

50 most cited of 367 All 367 (JSON)