# Improving LLM Leaderboards with Psychometrical Methodology

arXiv: https://arxiv.org/abs/2501.17200 · HTML: https://mlsanity.com/paper/2501.17200/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jan 2025.

## Built on by (most cited first, 5 of 5)

- [Rating the Raters: Rasch Measurement Theory for LLM Evaluation](https://mlsanity.com/paper/2608.27463/) (2026)
- [Contamination Inflates Scores but Rarely Reorders Large Language Model Leaderboards](https://mlsanity.com/paper/2609.02899/) (2026)
- [HumorRank: A Tournament-Based Leaderboard for Evaluating Humor Generation in Large Language Models](https://mlsanity.com/paper/2604.19786/) (2026)
- [Accounting for Bias Enables Sustainable LLM Evaluation](https://mlsanity.com/paper/2609.31184/) (2026)
- [Large-scale factor analysis shows machine intelligence is only partially interpretable](https://mlsanity.com/paper/2609.36515/) (2026)

## Abstract (arXiv)

The rapid development of large language models (LLMs) has necessitated the creation of benchmarks to evaluate their performance. These benchmarks resemble human tests and surveys, as they consist of sets of questions designed to measure emergent properties in the cognitive behavior of these systems. However, unlike the well-defined traits and abilities studied in social sciences, the properties measured by these benchmarks are often vaguer and less rigorously defined. The most prominent benchmarks are often grouped into leaderboards for convenience, aggregating performance metrics and enabling comparisons between models. Unfortunately, these leaderboards typically rely on simplistic aggregation methods, such as taking the average score across benchmarks. In this paper, we demonstrate the advantages of applying contemporary psychometric methodologies - originally developed for human tests and surveys - to improve the ranking of large language models on leaderboards. Using data from the Hugging Face Leaderboard as an example, we compare the results of the conventional naive ranking approach with a psychometrically informed ranking. The findings highlight the benefits of adopting psychometric techniques for more robust and meaningful evaluation of LLM performance.