# Benchmark Profiling: Mechanistic Diagnosis of LLM Benchmarks

arXiv: https://arxiv.org/abs/2510.01232 · HTML: https://mlsanity.com/paper/2510.01232/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 65 references that are in the map.
- First posted on arXiv 23 Sep 2025.

## Builds on (most cited first, 23 of 23)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [Importance Estimation for Neural Network Pruning](https://mlsanity.com/paper/1906.10771/) (2019)
- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [At Which Training Stage Does Code Data Help LLMs Reasoning?](https://mlsanity.com/paper/2309.16298/) (2023)
- [Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence](https://mlsanity.com/paper/2402.09880/) (2024)
- [Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation](https://mlsanity.com/paper/2502.06559/) (2025)
- [Unveiling the Impact of Coding Data Instruction Fine-Tuning on Large Language Models Reasoning](https://mlsanity.com/paper/2405.20535/) (2024)
- [Interpreting Arithmetic Mechanism in Large Language Models through Comparative Neuron Analysis](https://mlsanity.com/paper/2409.14144/) (2024)
- [The Vulnerability of Language Model Benchmarks: Do They Accurately Reflect True LLM Performance?](https://mlsanity.com/paper/2412.03597/) (2024)
- [When More is Less: Incorporating Additional Datasets Can Hurt Performance By Introducing Spurious Correlations](https://mlsanity.com/paper/2308.04431/) (2023)
- [The Generative AI Paradox on Evaluation: What It Can Solve, It May Not Evaluate](https://mlsanity.com/paper/2402.06204/) (2024)
- [Does External Knowledge Help Explainable Natural Language Inference? Automatic Evaluation vs. Human Ratings](https://mlsanity.com/paper/2109.07833/) (2021)

## Built on by (most cited first, 6 of 6)

- [Evaluation Cards: An Interpretive Layer for AI Evaluation Reporting](https://mlsanity.com/paper/2606.09809/) (2026)
- [Large Language Models for Agentic NetOps and AIOps: Architectures, Evaluation, and Safety](https://mlsanity.com/paper/2605.12729/) (2026)
- [Beyond Accuracy: Introducing a Symbolic-Mechanistic Approach to Interpretable Evaluation](https://mlsanity.com/paper/2603.23517/) (2026)
- [On the Measure of a Model: From Intelligence to Generality](https://mlsanity.com/paper/2511.11773/) (2025)
- [UniDial-EvalKit: A Unified Toolkit for Evaluating Multi-Faceted Conversational Abilities](https://mlsanity.com/paper/2603.23160/) (2026)
- [Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning](https://mlsanity.com/paper/2608.05139/) (2026)

## Abstract (arXiv)

Large Language Models are commonly judged by their scores on standard benchmarks, yet such scores often overstate real capability since they mask the mix of skills a task actually demands. For example, ARC is assumed to test reasoning, while HellaSwag is designed to evaluate commonsense. However, we lack a systematic way to verify if these benchmarks actually measure these labels. We introduce Benchmark Profiling, a diagnostic framework that decomposes benchmark performance into ten cognitively grounded abilities. The method combines gradient-based importance scoring with targeted parameter ablation to compute an Ability Impact Score (AIS) that quantifies how much each ability contributes to a model's success on a given benchmark. Profiling three instruction-tuned models across ten widely used benchmarks yields four key findings: (i) most benchmarks draw on several abilities rather than one, (ii) datasets with similar labels rely on distinct ability mixtures, (iii) code-generation benchmarks reward broad, multi-skill improvement and thus show only modest gains from narrow domain-specific fine-tuning, and (iv) abilities irrelevant to the task could negatively affect performance. Benchmark Profiling therefore explains why performance gains do not always translate into user-perceived competence and offers a transparent tool for benchmark audit and model interpretability.