Data to 7 Oct 2026

Do Large Language Model Benchmarks Test Reliability?

arXiv 2502.03461 ↗First posted 5 Feb 2025cs.LG cs.CLcited by 50

Not on the map. Why?

Abstract (arXiv)

When deploying large language models (LLMs), it is important to ensure that these models are not only capable, but also reliable. Many benchmarks have been created to track LLMs' growing capabilities, however there has been no similar focus on measuring their reliability. To understand the potential ramifications of this gap, we investigate how well current benchmarks quantify model reliability. We find that pervasive label errors can compromise these evaluations, obscuring lingering model failures and hiding unreliable behavior. Motivated by this gap in the evaluation of reliability, we then propose the concept of so-called platinum benchmarks, i.e., benchmarks carefully curated to minimize label errors and ambiguity. As a first attempt at constructing such benchmarks, we revise examples from fifteen existing popular benchmarks. We evaluate a wide range of models on these platinum benchmarks and find that, indeed, frontier LLMs still exhibit failures on simple tasks such as elementary-level math word problems. Analyzing these failures further reveals previously unidentified patterns of problems on which frontier models consistently struggle. We provide code at https://github.com/MadryLab/platinum-benchmarks

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Feb 2025.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 26; 2026: 24.

50 papers.

Sort:

By area:

2025Kimi K2: Open Agentic Intelligencecited by 523
2025The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMscited by 62
2025Rethinking Reflection in Pre-Trainingcited by 43
2025Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoningcited by 38
2025Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarkscited by 25
2025Antidistillation Samplingcited by 24
2026Noisy but Valid: Robust Statistical Evaluation of LLMs with Imperfect Judgescited by 23
2025Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluationcited by 22
2025Evaluating LLM Metrics Through Real-World Capabilitiescited by 17
2025Verifying the Verifiers: Unveiling Pitfalls and Potentials in Fact Verifierscited by 12
2025WUSH: Near-Optimal Adaptive Transforms for LLM Quantizationcited by 11
2025Scaling Graph Chain-of-Thought Reasoning: A Multi-Agent Framework with Efficient LLM Servingcited by 10

38 more

2026Benchmarks Saturate When The Model Gets Smarter Than The Judgecited by 10
2026Automated Benchmark Auditing for AI Agents and Large Language Modelscited by 10
2025Who Benchmarks the Benchmarks? Towards Comprehensive Evaluation of Commonsense Reasoning Benchmarkscited by 9
2025Min-p, Max Exaggeration: A Critical Analysis of Min-p Sampling in Language Modelscited by 9
2025Simple Policy Gradients for Reasoning with Diffusion Language Modelscited by 9
2025Fantastic Bugs and Where to Find Them in AI Benchmarkscited by 9
2025Benchmarking Misuse Mitigation Against Covert Adversariescited by 8
2025Position: Machine Learning Conferences Should Establish a "Refutations and Critiques" Trackcited by 7
2025CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfittingcited by 6
2025DESIGNER: Design-Logic-Guided Multidisciplinary Data Synthesis for LLM Reasoningcited by 6
2026Protecting Language Models Against Unauthorized Distillation through Trace Rewritingcited by 6
2026WildSci: Advancing Scientific Reasoning from In-the-Wild Literaturecited by 4
2026Weight Decay Improves Language Model Plasticitycited by 4
2026Beyond the Mean: Within-Model Reliable Change Detection for LLM Evaluationcited by 4
2025Learning a Continue-Thinking Token for Enhanced Test-Time Scalingcited by 3
2026Rethinking Math Reasoning Evaluation: A Robust LLM-as-a-Judge Framework Beyond Symbolic Rigiditycited by 3
2025Domain Specific Benchmarks for Evaluating Multimodal Large Language Modelscited by 2
2026Panini: Continual Learning in Token Space via Structured Memorycited by 2
2026Reasoning Core: A Scalable Procedural Data Generation Suite for Symbolic Pre-training and Post-Trainingcited by 2
2025Decoupling Understanding from Reasoning via Problem Space Mapping for Small-Scale Model Reasoningcited by 1
2026Measuring AI Reasoning: A Guide for Researcherscited by 1
2026Learning to Solve Hard Problems in RL for LLMs by Never Giving Upcited by 1
2025`A la recherche du sens perdu: your favourite LLM might have more to say than you can understandcited by 0
2025SWI: Speaking with Intent in Large Language Modelscited by 0
2025DETAIL Matters: Measuring the Impact of Prompt Specificity on Reasoning in Large Language Modelscited by 0
2025Automating Complex Document Workflows via Stepwise and Rollback-Enabled Operation Orchestrationcited by 0
2026Temporal Guidance for Large Language Modelscited by 0
2026LLM Olympiad: Why Model Evaluation Needs a Sealed Examcited by 0
2026Efficient Detection of Bad Benchmark Items with Novel Scalability Coefficientscited by 0
2026LLM Reasoning Is Latent, Not the Chain of Thoughtcited by 0
2026PathCal: State-Aware Reflection-Marker Calibration for Efficient Reasoningcited by 0
2026Learning What to Learn: Stage-Specific Data Sets for SFT-then-RL in Small Language Model Reasoningcited by 0
2026Flaws in the LLM Automation Narrativecited by 0
2026Answer-then-Edit: Reasoning Skeleton Editing for Anti-Distillation with Preserved Utilitycited by 0
2026Probing the Origins of Reasoning Performance: Representational Quality for Mathematical Problem-Solving in RL vs. SFT Fine-Tuned Modelscited by 0
2026HiRS-Agent: A Hierarchical Multi-Agent System for Reliable Long-Horizon Remote Sensing Task Solvingcited by 0
2026Loop Dropout: Regularizing Shared Updates in Looped Language Modelscited by 0
2026SERA: Scale-Equalized Rollout Allocation for Maximum Likelihood Reinforcement Learningcited by 0