mlsanityData to 7 Oct 2026

Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference

arXiv 2403.04132 ↗First posted 7 Mar 2024cs.AI cs.CLcited by 1,151

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Large Language Models (LLMs) have unlocked new capabilities and applications; however, evaluating the alignment with human preferences still poses significant challenges. To address this issue, we introduce Chatbot Arena, an open platform for evaluating LLMs based on human preferences. Our methodology employs a pairwise comparison approach and leverages input from a diverse user base through crowdsourcing. The platform has been operational for several months, amassing over 240K votes. This paper describes the platform, analyzes the data we have collected so far, and explains the tried-and-true statistical methods we are using for efficient and accurate evaluation and ranking of models. We confirm that the crowdsourced questions are sufficiently diverse and discriminating and that the crowdsourced human votes are in good agreement with those of expert raters. These analyses collectively establish a robust foundation for the credibility of Chatbot Arena. Because of its unique value and openness, Chatbot Arena has emerged as one of the most referenced LLM leaderboards, widely cited by leading LLM developers and companies. Our demo is publicly available at urlhttps://chat.lmsys.org.

Key facts. Cited by 1,151 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 25 of its 71 references that are in the map. First posted on arXiv 7 Mar 2024.

Builds on

25 of 71 references.

References · 25

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  11. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  12. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
13 more
  1. 2022BERTopic: Neural topic modeling with a class-based TF-IDF procedurecited by 779
  2. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  3. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  4. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  5. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  6. 2023Proving Test Set Contamination in Black Box Language Modelscited by 227
  7. 2023ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversationcited by 224
  8. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 192
  9. 2020Estimating means of bounded random variables by bettingcited by 164
  10. 2022Game-theoretic statistics and safe anytime-valid inferencecited by 145
  11. 2021The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generationcited by 96
  12. 2023Competition-Level Problems are Effective LLM Evaluatorscited by 25
  13. 2020Online Active Model Selection for Pre-trained Classifierscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 270; 2025: 479; 2026: 402.

1,151 papers.

Sort:

By area:

  1. 2024Qwen2 Technical Reportcited by 2,076
  2. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  3. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  4. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  5. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  6. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  7. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  8. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMscited by 424
  9. 2025LLMs Get Lost In Multi-Turn Conversationcited by 400
  10. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 344
  11. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  12. 2026SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Taskscited by 289
38 more
  1. 2026Reinforcement Learning via Self-Distillationcited by 285
  2. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Modelscited by 277
  3. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 242
  4. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
  5. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 224
  6. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 197
  7. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 138
  8. 2024Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluationscited by 130
  9. 2024AudioBench: A Universal Benchmark for Audio Large Language Modelscited by 124
  10. 2024Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agentscited by 120
  11. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 118
  12. 2025WorldModelBench: Judging Video Generation Models As World Modelscited by 116
  13. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 112
  14. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  15. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  16. 2025One-Minute Video Generation with Test-Time Trainingcited by 100
  17. 2024Chain of Ideas: Revolutionizing Research Via Novel Idea Development with LLM Agentscited by 94
  18. 2024GameBench: Evaluating Strategic Reasoning Abilities of LLM Agentscited by 89
  19. 2025MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Serverscited by 89
  20. 2024What If We Recaption Billions of Web Images with LLaMA-3?cited by 82
  21. 2024CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understandingcited by 82
  22. 2024GenAI Arena: An Open Evaluation Platform for Generative Modelscited by 81
  23. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safetycited by 75
  24. 2024MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixturescited by 73
  25. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUscited by 73
  26. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 73
  27. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 72
  28. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 71
  29. 2024Reinforcement Learning Enhanced LLMs: A Surveycited by 69
  30. 2024FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructionscited by 67
  31. 2024Adaptive Inference-Time Compute: LLMs Can Predict if They Can Do Better, Even Mid-Generationcited by 64
  32. 2025RoboArena: Distributed Real-World Evaluation of Generalist Robot Policiescited by 63
  33. 2024Archon: An Architecture Search Framework for Inference-Time Techniquescited by 59
  34. 2025Large language models for automated scholarly paper review: A surveycited by 59
  35. 2025ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decompositioncited by 57
  36. 2024Large Language Models Can Self-Improve At Web Agent Taskscited by 54
  37. 2024AutoEval Done Right: Using Synthetic Data for Model Evaluationcited by 53
  38. 2025fev-bench: A Realistic Benchmark for Time Series Forecastingcited by 52

50 most cited of 1,151 All 1,151 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023TencentLLMEval: A Hierarchical Evaluation of Real-World Capabilities for Human-Aligned LLMs
  2. 2024Dissecting Human and LLM Preferences
  3. 2024Testing and Evaluation of Large Language Models: Correctness, Non-Toxicity, and Fairness
  4. 2024Self-Preference Bias in LLM-as-a-Judge
  5. 2024Evaluating Large Language Models with Human Feedback: Establishing a Swedish Benchmark
  6. 2025MEDAL: A Framework for Benchmarking LLMs as Multilingual Open-Domain Dialogue Evaluators
  7. 2025Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgments
  8. 2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs