mlsanityData to 14 Oct 2026

WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild

arXiv 2406.04770 ↗First posted 7 Jun 2024cs.CL cs.AIcited by 218

Not on the map. Why?

Cited by 33 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce WildBench, an automated evaluation framework designed to benchmark large language models (LLMs) using challenging, real-world user queries. WildBench consists of 1,024 tasks carefully selected from over one million human-chatbot conversation logs. For automated evaluation with WildBench, we have developed two metrics, WB-Reward and WB-Score, which are computable using advanced LLMs such as GPT-4-turbo. WildBench evaluation uses task-specific checklists to evaluate model outputs systematically and provides structured explanations that justify the scores and comparisons, resulting in more reliable and interpretable automatic judgments. WB-Reward employs fine-grained pairwise comparisons between model responses, generating five potential outcomes: much better, slightly better, slightly worse, much worse, or a tie. Unlike previous evaluations that employed a single baseline model, we selected three baseline models at varying performance levels to ensure a comprehensive pairwise evaluation. Additionally, we propose a simple method to mitigate length bias, by converting outcomes of ``slightly better/worse'' to ``tie'' if the winner response exceeds the loser one by more than K characters. WB-Score evaluates the quality of model outputs individually, making it a fast and cost-efficient evaluation metric. WildBench results demonstrate a strong correlation with the human-voted Elo ratings from Chatbot Arena on hard tasks. Specifically, WB-Reward achieves a Pearson correlation of 0.98 with top-ranking models. Additionally, WB-Score reaches 0.95, surpassing both ArenaHard's 0.91 and AlpacaEval2.0's 0.89 for length-controlled win rates, as well as the 0.87 for regular win rates.

Key facts. Cited by 218 papers in the map (data through 14 Oct 2026). Builds on 21 of its 35 references that are in the map. First posted on arXiv 7 Jun 2024.

Builds on

21 of 35 references.

References · 21

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  8. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  9. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  10. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  11. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  12. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
9 more
  1. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  2. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  3. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
  4. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  5. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 257
  6. 2024KMMLU: Measuring Massive Multitask Language Understanding in Koreancited by 91
  7. 2023TIGERScore: Towards Building Explainable Metric for All Text Generation Taskscited by 70
  8. 2023The Shifted and The Overlooked: A Task-oriented Investigation of User-GPT Interactionscited by 45
  9. 2022On Grounded Planning for Embodied Tasks with Language Modelscited by 33

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 35; 2025: 81; 2026: 102.

218 papers.

Sort:

By area:

  1. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  2. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  3. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
  4. 2025Why Language Models Hallucinatecited by 220
  5. 2025SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilitiescited by 136
  6. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 132
  7. 2025xbench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluationscited by 120
  8. 2024The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinismcited by 116
  9. 2026Ministral 3cited by 114
  10. 2025UserBench: An Interactive Gym Environment for User-Centric Agentscited by 76
  11. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languagescited by 75
  12. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 74
38 more
  1. 2024TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generationcited by 68
  2. 2025No Free Labels: Limitations of LLM-as-a-Judge Without Human Groundingcited by 54
  3. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 54
  4. 2024Jamba-1.5: Hybrid Transformer-Mamba Models at Scalecited by 50
  5. 2025ResearchQA: Evaluating Scholarly Question Answering at Scale Across 75 Fields with Survey-Mined Questions and Rubricscited by 49
  6. 2024Diverging Preferences: When do Annotators Disagree and do Models Know?cited by 47
  7. 2025Toward an Evaluation Science for Generative AI Systemscited by 45
  8. 2024HelloBench: Evaluating Long Text Generation Capabilities of Large Language Modelscited by 44
  9. 2024CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolutioncited by 42
  10. 2025CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Eracited by 41
  11. 2025RocketEval: Efficient Automated LLM Evaluation via Grading Checklistcited by 39
  12. 2025Improving LLM-as-a-Judge Inference with the Judgment Distributioncited by 37
  13. 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Modelscited by 35
  14. 2025Investigating Non-Transitivity in LLM-as-a-Judgecited by 34
  15. 2025ChatBench: From Static Benchmarks to Human-AI Evaluationcited by 34
  16. 2025R-Bench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluationcited by 34
  17. 2025Search Arena: Analyzing Search-Augmented LLMscited by 34
  18. 2026DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reportscited by 34
  19. 2024LMUnit: Fine-grained Evaluation with Natural Language Unit Testscited by 33
  20. 2026Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environmentscited by 33
  21. 2026STEP3-VL-10B Technical Reportcited by 32
  22. 2024WildFeedback: Aligning LLMs With In-situ User Interactions And Feedbackcited by 31
  23. 2025Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoningcited by 29
  24. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 28
  25. 2026Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Biascited by 28
  26. 2026Gym-Anything: Turn any Software into an Agent Environmentcited by 27
  27. 2024A Benchmark for Long-Form Medical Question Answeringcited by 25
  28. 2025Genius: A Generalizable and Purely Unsupervised Self-Training Framework For Advanced Reasoningcited by 24
  29. 2025BOLT: Bootstrap Long Chain-of-Thought in Language Models without Distillationcited by 23
  30. 2025R3: Robust Rubric-Agnostic Reward Modelscited by 23
  31. 2024ReIFE: Re-evaluating Instruction-Following Evaluationcited by 22
  32. 2024CARMO: Dynamic Criteria Generation for Context-Aware Reward Modellingcited by 22
  33. 2025CUPID: Evaluating Personalized and Contextualized Alignment of LLMs from Interactionscited by 22
  34. 2024EXAONE 3.0 7.8B Instruction Tuned Language Modelcited by 21
  35. 2025From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centeredcited by 21
  36. 2025MoNaCo: More Natural and Complex Questions for Reasoning Across Dozens of Documentscited by 21
  37. 2025Kanana: Compute-efficient Bilingual Language Modelscited by 20
  38. 2024RAD-Bench: Evaluating Large Language Models Capabilities in Retrieval Augmented Dialoguescited by 19

50 most cited of 218 All 218 (JSON)