mlsanityData to 7 Oct 2026

MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures

arXiv 2406.06565 ↗First posted 3 Jun 2024cs.CL cs.AI cs.LGcited by 73

Not on the map. Why?

Abstract (arXiv)

Evaluating large language models (LLMs) is challenging. Traditional ground-truth-based benchmarks fail to capture the comprehensiveness and nuance of real-world queries, while LLM-as-judge benchmarks suffer from grading biases and limited query quantity. Both of them may also become contaminated over time. User-facing evaluation, such as Chatbot Arena, provides reliable signals but is costly and slow. In this work, we propose MixEval, a new paradigm for establishing efficient, gold-standard LLM evaluation by strategically mixing off-the-shelf benchmarks. It bridges (1) comprehensive and well-distributed real-world user queries and (2) efficient and fairly-graded ground-truth-based benchmarks, by matching queries mined from the web with similar queries from existing benchmarks. Based on MixEval, we further build MixEval-Hard, which offers more room for model improvement. Our benchmarks' advantages lie in (1) a 0.96 model ranking correlation with Chatbot Arena arising from the highly impartial query distribution and grading mechanism, (2) fast, cheap, and reproducible execution (6% of the time and cost of MMLU), and (3) dynamic evaluation enabled by the rapid and stable data update pipeline. We provide extensive meta-evaluation and analysis for our and existing LLM benchmarks to deepen the community's understanding of LLM evaluation and guide future research directions.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Builds on 31 of its 49 references that are in the map. First posted on arXiv 3 Jun 2024.

Builds on

31 of 49 references.

References · 31

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2021Program Synthesis with Large Language Modelscited by 3,509
  10. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  12. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
19 more
  1. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  2. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  3. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  5. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  6. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  7. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  8. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  9. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  10. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  11. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  12. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  13. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 192
  14. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 188
  15. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Rewardcited by 145
  16. 2024MAmmoTH2: Scaling Instructions from the Webcited by 143
  17. 2023NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classescited by 78
  18. 2024Compression Represents Intelligence Linearlycited by 56
  19. 2024Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language modelscited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 26; 2025: 35; 2026: 12.

73 papers.

Sort:

By area:

  1. 2024Qwen2 Technical Reportcited by 2,076
  2. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  3. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 119
  4. 2024The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinismcited by 113
  5. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
  6. 2024A Unified Approach to Routing and Cascading for LLMscited by 71
  7. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignmentcited by 60
  8. 2024Archon: An Architecture Search Framework for Inference-Time Techniquescited by 59
  9. 2025ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decompositioncited by 57
  10. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignmentcited by 46
  11. 2024Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Ratescited by 42
  12. 2025RocketEval: Efficient Automated LLM Evaluation via Grading Checklistcited by 37
38 more
  1. 2025CARROT: A Cost Aware Rate Optimal Routercited by 34
  2. 2024Evaluating Language Models as Synthetic Data Generatorscited by 32
  3. 2025Fluid Language Model Benchmarkingcited by 30
  4. 2024Meta Reasoning for Large Language Modelscited by 29
  5. 2024SPRIG: Improving Large Language Model Performance by System Prompt Optimizationcited by 28
  6. 2025When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Researchcited by 25
  7. 2024100 instances is all you need: predicting the success of a new LLM on unseen data by testing on a few instancescited by 24
  8. 2025How to Select Datapoints for Efficient Human Evaluation of NLG Models?cited by 23
  9. 2024Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignmentcited by 20
  10. 2024Improving Model Evaluation using SMART Filtering of Benchmark Datasetscited by 20
  11. 2024Re-evaluating Automatic LLM System Ranking for Alignment with Human Preferencecited by 16
  12. 2025CondAmbigQA: A Benchmark and Dataset for Conditional Ambiguous Question Answeringcited by 15
  13. 2024MIRAGE-Bench: Automatic Multilingual Benchmark Arena for Retrieval-Augmented Generation Systemscited by 14
  14. 2025The Emperor's New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contaminationcited by 13
  15. 2025NarraBench: A Comprehensive Framework for Narrative Benchmarkingcited by 13
  16. 2024FB-Bench: A Fine-Grained Multi-Task Benchmark for Evaluating LLMs' Responsiveness to Human Feedbackcited by 11
  17. 2025Tuning LLM Judge Design Decisions for 1/1000 of the Costcited by 11
  18. 2024LIME: Less Is More for MLLM Evaluationcited by 7
  19. 2025LLM Alignment as Retriever Optimization: An Information Retrieval Perspectivecited by 7
  20. 2025Post-training for Efficient Communication via Convention Formationcited by 7
  21. 2024MixEval-X: Any-to-Any Evaluations from Real-World Data Mixturescited by 6
  22. 2024Baichuan Alignment Technical Reportcited by 6
  23. 2025CrowdSelect: Synthetic Instruction Data Selection with Multi-LLM Wisdomcited by 6
  24. 2025Estimating Machine Translation Difficultycited by 6
  25. 2025ReasonBENCH: Benchmarking the (In)Stability of LLM Reasoningcited by 6
  26. 2025MMMG: a Comprehensive and Reliable Benchmark for Multitask Multimodal Generationcited by 5
  27. 2024Why We Build Local Large Language Models: An Observational Analysis from 35 Japanese and Multilingual LLMscited by 4
  28. 2025WILDCHAT-50M: A Deep Dive Into the Role of Synthetic Data in Post-Trainingcited by 4
  29. 2025Unnatural Languages Are Not Bugs but Features for LLMscited by 4
  30. 2025Bielik v3 Small: Technical Reportcited by 4
  31. 2025When Instructions Multiply: Measuring and Estimating LLM Capabilities of Multiple Instructions Followingcited by 4
  32. 2024Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluationcited by 3
  33. 2025BenchHub: A Unified Benchmark Suite for Holistic and Customizable LLM Evaluationcited by 3
  34. 2025Establishing Trustworthy LLM Evaluation via Shortcut Neuron Analysiscited by 3
  35. 2025DCRM: A Heuristic to Measure Response Pair Quality in Preference Optimizationcited by 3
  36. 2025Lean Meets Theoretical Computer Science: Scalable Synthesis of Theorem Proving Challenges in Formal-Informal Pairscited by 3
  37. 2025On Evaluating LLM Alignment by Evaluating LLMs as Judgescited by 3
  38. 2026Breaking Model Lock-in: Cost-Efficient Zero-Shot LLM Routing via a Universal Latent Spacecited by 3

50 most cited of 73 All 73 (JSON)