mlsanityData to 14 Oct 2026

LiveBench: A Challenging, Contamination-Limited LLM Benchmark

arXiv 2406.19314 ↗First posted 27 Jun 2024cs.CL cs.AI cs.LGcited by 262

Not on the map. Why?

Cited by 61 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Test set contamination, wherein test data from a benchmark ends up in a newer model's training set, is a well-documented obstacle for fair LLM evaluation and can quickly render benchmarks obsolete. To mitigate this, many recent benchmarks crowdsource new prompts and evaluations from human or LLM judges; however, these can introduce significant biases, and break down when scoring hard questions. In this work, we introduce a new benchmark for LLMs designed to be resistant to both test set contamination and the pitfalls of LLM judging and human crowdsourcing. We release LiveBench, the first benchmark that (1) contains frequently-updated questions from recent information sources, (2) scores answers automatically according to objective ground-truth values, and (3) contains a wide variety of challenging tasks, spanning math, coding, reasoning, language, instruction following, and data analysis. To achieve this, LiveBench contains questions that are based on recently-released math competitions, arXiv papers, news articles, and datasets, and it contains harder, contamination-limited versions of tasks from previous benchmarks such as Big-Bench Hard, AMPS, and IFEval. We evaluate many prominent closed-source models, as well as dozens of open-source models ranging from 0.5B to 405B in size. LiveBench is difficult, with top models achieving below 70% accuracy. We release all questions, code, and model answers. Questions are added and updated on a monthly basis, and we release new tasks and harder versions of tasks over time so that LiveBench can distinguish between the capabilities of LLMs as they improve in the future. We welcome community engagement and collaboration for expanding the benchmark tasks and models.

Key facts. Cited by 262 papers in the map (data through 14 Oct 2026). Builds on 37 of its 69 references that are in the map. First posted on arXiv 27 Jun 2024.

Builds on

37 of 69 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  10. 2023Qwen Technical Reportcited by 3,344
  11. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  12. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
25 more
  1. 2023Mistral 7Bcited by 3,164
  2. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  3. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  4. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  5. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  6. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  7. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  8. 2018Datasheets for Datasetscited by 1,386
  9. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  10. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  11. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  12. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  13. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  14. 2023StarCoder: may the source be with you!cited by 799
  15. 2021Measuring Attribution in Natural Language Generation Modelscited by 256
  16. 2023Proving Test Set Contamination in Black Box Language Modelscited by 230
  17. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 191
  18. 2023Time Travel in LLMs: Tracing Data Contamination in Large Language Modelscited by 177
  19. 2024Nemotron-4 340B Technical Reportcited by 144
  20. 2023Investigating Data Contamination in Modern Benchmarks for Large Language Modelscited by 138
  21. 2024Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Modelscited by 122
  22. 2024Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gapcited by 79
  23. 2023Data Contamination Quiz: A Tool to Detect and Estimate Contamination in Large Language Modelscited by 47
  24. 2024ReMatch: Retrieval Enhanced Schema Matching with LLMscited by 16
  25. 2023ArcheType: A Novel Framework for Open-Source Column Type Annotation using Large Language Modelscited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 78; 2026: 174.

262 papers.

Sort:

By area:

  1. 2024Qwen2.5 Technical Reportcited by 4,942
  2. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  3. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMscited by 289
  4. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 214
  5. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  6. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 170
  7. 2025Measuring what Matters: Construct Validity in Large Language Model Benchmarkscited by 119
  8. 2025The Invisible Leash: Why RLVR May or May Not Escape Its Origincited by 97
  9. 2025AdaCoT: Pareto-Optimal Adaptive Chain-of-Thought Triggering via Reinforcement Learningcited by 87
  10. 2025CodeI/O: Condensing Reasoning Patterns via Code Input-Output Predictioncited by 55
  11. 2025UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Modelscited by 48
  12. 2025Model Merging in Pre-training of Large Language Modelscited by 37
38 more
  1. 2025LLMs Cannot Reliably Judge (Yet?): A Comprehensive Assessment on the Robustness of LLM-as-a-Judgecited by 37
  2. 2025A Rosetta Stone for AI Benchmarkscited by 36
  3. 2025Investigating Non-Transitivity in LLM-as-a-Judgecited by 34
  4. 2026When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturationcited by 33
  5. 2024LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrationscited by 32
  6. 2024GameArena: Evaluating LLM Reasoning through Live Computer Gamescited by 32
  7. 2025AutoLogi: Automated Generation of Logic Puzzles for Evaluating Reasoning Abilities of Large Language Modelscited by 31
  8. 2025Nuclear Deployed: Analyzing Catastrophic Risks in Decision-making of Autonomous LLM Agentscited by 27
  9. 2025CodeRL+: Improving Code Generation via Reinforcement with Execution Semantics Alignmentcited by 24
  10. 2025Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contaminationcited by 23
  11. 2025DyFlow: Dynamic Workflow Framework for Agentic Reasoningcited by 23
  12. 2025Learning to Orchestrate Agents in Natural Language with the Conductorcited by 23
  13. 2026Large Language Model Reasoning Failurescited by 21
  14. 2025LLM-Safety Evaluations Lack Robustnesscited by 18
  15. 2025Towards Concise and Adaptive Thinking in Large Reasoning Models: A Surveycited by 18
  16. 2024GRIN: GRadient-INformed MoEcited by 17
  17. 2025FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documentscited by 15
  18. 2026Counteraction-Aware Multi-Teacher On-Policy Distillation for General Capability Recovery with Domain Preservationcited by 15
  19. 2024Refusal Tokens: A Simple Way to Calibrate Refusals in Large Language Modelscited by 14
  20. 2024GLIDER: Grading LLM Interactions and Decisions using Explainable Rankingcited by 14
  21. 2025LastingBench: Defend Benchmarks Against Knowledge Leakagecited by 14
  22. 2024MathHay: An Automated Benchmark for Long-Context Mathematical Reasoning in LLMscited by 13
  23. 2025Towards Reliable Benchmarking: A Contamination Free, Controllable Evaluation Framework for Multi-step LLM Function Callingcited by 13
  24. 2026OneMillion-Bench: How Far are Language Agents from Human Experts?cited by 13
  25. 2026AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks?cited by 13
  26. 2025Thinking Out Loud: Do Reasoning Models Know When They're Right?cited by 12
  27. 2025SABER: Switchable and Balanced Training for Efficient LLM Reasoningcited by 12
  28. 2024Evaluation-Driven Development and Operations of LLM Agents: A Process Model and Reference Architecturecited by 11
  29. 2025CipherBank: Exploring the Boundary of LLM Reasoning Capabilities through Cryptography Challengescited by 11
  30. 2025General Modular Harness for LLM Agents in Multi-Turn Gaming Environmentscited by 11
  31. 2025MoHoBench: Assessing Honesty of Multimodal Large Language Models via Unanswerable Visual Questionscited by 11
  32. 2026Behavior Knowledge Merge in Reinforced Agentic Modelscited by 11
  33. 2026Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillationcited by 11
  34. 2025Why Tree-Style Branching Matters for Thought Advantage Estimation in GRPOcited by 10
  35. 2025ARC-GEN: A Mimetic Procedural Benchmark Generator for the Abstraction and Reasoning Corpuscited by 10
  36. 2026ProbeLLM: Automating Principled Diagnosis of LLM Failurescited by 9
  37. 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gapcited by 9
  38. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resourcecited by 8

50 most cited of 262 All 262 (JSON)