mlsanityData to 7 Oct 2026

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

arXiv 2306.05685 ↗First posted 9 Jun 2023cs.CL cs.AIcited by 8,393

Not on the map. Why?

Cited by 1,167 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Evaluating large language model (LLM) based chat assistants is challenging due to their broad capabilities and the inadequacy of existing benchmarks in measuring human preferences. To address this, we explore using strong LLMs as judges to evaluate these models on more open-ended questions. We examine the usage and limitations of LLM-as-a-judge, including position, verbosity, and self-enhancement biases, as well as limited reasoning ability, and propose solutions to mitigate some of them. We then verify the agreement between LLM judges and human preferences by introducing two benchmarks: MT-bench, a multi-turn question set; and Chatbot Arena, a crowdsourced battle platform. Our results reveal that strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans. Hence, LLM-as-a-judge is a scalable and explainable way to approximate human preferences, which are otherwise very expensive to obtain. Additionally, we show our benchmark and traditional benchmarks complement each other by evaluating several variants of LLaMA and Vicuna. The MT-bench questions, 3K expert votes, and 30K conversations with human preferences are publicly available at https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge.

Key facts. Cited by 8,393 papers in the map (data through 7 Oct 2026). Builds on 36 of its 59 references that are in the map. First posted on arXiv 9 Jun 2023.

Builds on

36 of 59 references.

References · 36

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  11. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  12. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
24 more
  1. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  3. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  4. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  5. 2023PaLM 2 Technical Reportcited by 1,154
  6. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  7. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  8. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  9. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  10. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  11. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  12. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  13. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  14. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  15. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  16. 2023Instruction Tuning with GPT-4cited by 676
  17. 2018CoQA: A Conversational Question Answering Challengecited by 674
  18. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  19. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  20. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  21. 2023The False Promise of Imitating Proprietary LLMscited by 254
  22. 2023Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speechcited by 165
  23. 2020Look at the First Sentence: Position Bias in Question Answeringcited by 83
  24. 2022MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversationcited by 56

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 455; 2024: 2099; 2025: 2508; 2026: 3331.

8,393 papers.

Sort:

By area:

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  5. 2024Qwen2 Technical Reportcited by 2,076
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  7. 2024Mixtral of Expertscited by 1,785
  8. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  9. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  10. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,411
  11. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  12. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
38 more
  1. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  2. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  3. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  4. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  5. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  6. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  7. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  8. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  9. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  10. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  11. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  12. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  13. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  14. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  15. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  16. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  17. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  18. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  19. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  20. 2024Yi: Open Foundation Models by 01.AIcited by 706
  21. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  22. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  23. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  24. 2024Self-Rewarding Language Modelscited by 654
  25. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  26. 2024A Survey on Large Language Models for Code Generationcited by 646
  27. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  28. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  29. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  30. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  31. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  32. 2025Why Do Multi-Agent LLM Systems Fail?cited by 559
  33. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  34. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  35. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  36. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  37. 2024Executable Code Actions Elicit Better LLM Agentscited by 537
  38. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529

50 most cited of 8,393 All 8,393 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Are We on the Right Way to Assessing LLM-as-a-Judge?
  2. 2025Improve LLM-as-a-Judge Ability as a General Ability
  3. 2024GameArena: Evaluating LLM Reasoning through Live Computer Games
  4. 2024DebateQA: Evaluating Question Answering on Debatable Knowledge
  5. 2024On the Benchmarking of LLMs for Open-Domain Dialogue Evaluation
  6. 2024From Calculation to Adjudication: Examining LLM judges on Mathematical Reasoning Tasks
  7. 2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs
  8. 2024Evaluating Large Language Models with Human Feedback: Establishing a Swedish Benchmark