mlsanityData to 13 Oct 2026

From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline

arXiv 2406.11939 ↗First posted 17 Jun 2024cs.LG cs.AI cs.CLcited by 491

Not on the map. Why?

Cited by 153 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The rapid evolution of Large Language Models (LLMs) has outpaced the development of model evaluation, highlighting the need for continuous curation of new, challenging benchmarks. However, manual curation of high-quality, human-aligned benchmarks is expensive and time-consuming. To address this, we introduce BenchBuilder, an automated pipeline that leverages LLMs to curate high-quality, open-ended prompts from large, crowd-sourced datasets, enabling continuous benchmark updates without human in the loop. We apply BenchBuilder to datasets such as Chatbot Arena and WildChat-1M, extracting challenging prompts and utilizing LLM-as-a-Judge for automatic model evaluation. To validate benchmark quality, we propose new metrics to measure a benchmark's alignment with human preferences and ability to separate models. We release Arena-Hard-Auto, a benchmark consisting 500 challenging prompts curated by BenchBuilder. Arena-Hard-Auto provides 3x higher separation of model performances compared to MT-Bench and achieves 98.6% correlation with human preference rankings, all at a cost of 20. Our work sets a new framework for the scalable curation of automated benchmarks from extensive data.

Key facts. Cited by 491 papers in the map (data through 13 Oct 2026). Builds on 32 of its 66 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

32 of 66 references.

References · 32

  1. 2023GPT-4 Technical Reportcited by 18,278
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,916
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,205
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,876
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,270
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,880
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,133
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,053
  11. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,274
  12. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,197
20 more
  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,693
  2. 2020Extracting Training Data from Large Language Modelscited by 2,181
  3. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,985
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,951
  5. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,908
  6. 2015A large annotated corpus for learning natural language inferencecited by 1,841
  7. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,191
  8. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 890
  9. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 877
  10. 2024Yi: Open Foundation Models by 01.AIcited by 715
  11. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 705
  12. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 616
  13. 2021Dynabench: Rethinking Benchmarking in NLPcited by 452
  14. 2024InternLM2 Technical Reportcited by 393
  15. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 347
  16. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 313
  17. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 212
  18. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 194
  19. 2021The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generationcited by 96
  20. 2024MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixturescited by 73

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 74; 2025: 230; 2026: 187.

491 papers.

Sort:

By area:

  1. 2025Qwen3 Technical Reportcited by 8,510
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,247
  3. 2024Qwen2.5 Technical Reportcited by 4,840
  4. 2025Qwen3-VL Technical Reportcited by 2,663
  5. 2024Qwen2 Technical Reportcited by 2,102
  6. 2024DeepSeek-V3 Technical Reportcited by 777
  7. 2025Kimi K2: Open Agentic Intelligencecited by 544
  8. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 483
  9. 2026Reinforcement Learning via Self-Distillationcited by 310
  10. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 304
  11. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 275
  12. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 220
38 more
  1. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 203
  2. 2025Qwen2.5-1M Technical Reportcited by 159
  3. 2025Phi-4-reasoning Technical Reportcited by 147
  4. 2025Qwen3Guard Technical Reportcited by 147
  5. 2024Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontiercited by 128
  6. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 121
  7. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 116
  8. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 108
  9. 2024Critique-out-Loud Reward Modelscited by 105
  10. 2026Ministral 3cited by 103
  11. 2025WritingBench: A Comprehensive Benchmark for Generative Writingcited by 98
  12. 2024Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencentcited by 90
  13. 2025CWM: An Open-Weights LLM for Research on Code Generation with World Modelscited by 90
  14. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 89
  15. 2024How to Evaluate Reward Models for RLHFcited by 84
  16. 2025Universal Model Routing for Efficient LLM Inferencecited by 82
  17. 2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 80
  18. 2025TreeRL: LLM Reinforcement Learning with On-Policy Tree Searchcited by 76
  19. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 72
  20. 2025Think Only When You Need with Large Hybrid-Reasoning Modelscited by 64
  21. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignmentcited by 60
  22. 2025Baichuan-M2: Scaling Medical Capability with Large Verifier Systemcited by 57
  23. 2026PostTrainBench: Can LLM Agents Automate LLM Post-Training?cited by 56
  24. 2025LongCat-Flash Technical Reportcited by 55
  25. 2025rStar2-Agent: Agentic Reasoning Technical Reportcited by 54
  26. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 52
  27. 2024EXAONE 3.5: Series of Large Language Models for Real-world Use Casescited by 51
  28. 2025No Free Labels: Limitations of LLM-as-a-Judge Without Human Groundingcited by 50
  29. 2024Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the datacited by 49
  30. 2025AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Followingcited by 49
  31. 2024Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklistcited by 48
  32. 2025Reward Reasoning Modelcited by 47
  33. 2026RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generationcited by 47
  34. 2024EmbedLLM: Learning Compact Representations of Large Language Modelscited by 46
  35. 2024Thinking LLMs: General Instruction Following with Thought Generationcited by 45
  36. 2025Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Modelscited by 45
  37. 2024Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Ratescited by 44
  38. 2025AI Flow: Perspectives, Scenarios, and Approachescited by 44

50 most cited of 491 All 491 (JSON)