# Utility is in the Eye of the User: A Critique of NLP Leaderboards

arXiv: https://arxiv.org/abs/2009.13888 · HTML: https://mlsanity.com/paper/2009.13888/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Sep 2020.

## Built on by (most cited first, 28 of 28)

- [Dynabench: Rethinking Benchmarking in NLP](https://mlsanity.com/paper/2104.14337/) (2021)
- [AI and the Everything in the Whole Wide World Benchmark](https://mlsanity.com/paper/2111.15366/) (2021)
- [The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics](https://mlsanity.com/paper/2102.01672/) (2021)
- [What Will it Take to Fix Benchmarking in Natural Language Understanding?](https://mlsanity.com/paper/2104.02145/) (2021)
- [A Roadmap to Pluralistic Alignment](https://mlsanity.com/paper/2402.05070/) (2024)
- [The Values Encoded in Machine Learning Research](https://mlsanity.com/paper/2106.15590/) (2021)
- [XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation](https://mlsanity.com/paper/2104.07412/) (2021)
- [Survey of Low-Resource Machine Translation](https://mlsanity.com/paper/2109.00486/) (2021)
- [Is Automated Topic Model Evaluation Broken?: The Incoherence of Coherence](https://mlsanity.com/paper/2107.02173/) (2021)
- [Problems with Cosine as a Measure of Embedding Similarity for High Frequency Words](https://mlsanity.com/paper/2205.05092/) (2022)
- [Evaluation Gaps in Machine Learning Practice](https://mlsanity.com/paper/2205.05256/) (2022)
- [Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold](https://mlsanity.com/paper/2206.09755/) (2022)
- [Inherent Trade-Offs between Diversity and Stability in Multi-Task Benchmarks](https://mlsanity.com/paper/2405.01719/) (2024)
- [Knowledge Graph Question Answering Leaderboard: A Community Resource to Prevent a Replication Crisis](https://mlsanity.com/paper/2201.08174/) (2022)
- [This is the way: designing and compiling LEPISZCZE, a comprehensive NLP benchmark for Polish](https://mlsanity.com/paper/2211.13112/) (2022)
- [How not to Lie with a Benchmark: Rearranging NLP Leaderboards](https://mlsanity.com/paper/2112.01342/) (2021)
- [Mukayese: Turkish NLP Strikes Back](https://mlsanity.com/paper/2203.01215/) (2022)
- [Computational Hermeneutics: Evaluating generative AI as a cultural technology](https://mlsanity.com/paper/2604.16403/) (2026)
- [Richer Countries and Richer Representations](https://mlsanity.com/paper/2205.05093/) (2022)
- [STEER-ME: Assessing the Microeconomic Reasoning of Large Language Models](https://mlsanity.com/paper/2502.13119/) (2025)
- [Expected Validation Performance and Estimation of a Random Variable's Maximum](https://mlsanity.com/paper/2110.00613/) (2021)
- [bgGLUE: A Bulgarian General Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2306.02349/) (2023)
- [Challenges for cognitive decoding using deep learning methods](https://mlsanity.com/paper/2108.06896/) (2021)
- [Beyond Accuracy: A Consolidated Tool for Visual Question Answering Benchmarking](https://mlsanity.com/paper/2110.05159/) (2021)
- [Dim Wihl Gat Tun: The Case for Linguistic Expertise in NLP for Underdocumented Languages](https://mlsanity.com/paper/2203.09632/) (2022)
- [Unreasonable Effectiveness of Rule-Based Heuristics in Solving Russian SuperGLUE Tasks](https://mlsanity.com/paper/2105.01192/) (2021)
- [A Discussion on Building Practical NLP Leaderboards: The Case of Machine Translation](https://mlsanity.com/paper/2106.06292/) (2021)
- [An Interdisciplinary Perspective on Evaluation and Experimental Design for Visual Text Analytics: Position Paper](https://mlsanity.com/paper/2209.11534/) (2022)

## Abstract (arXiv)

Benchmarks such as GLUE have helped drive advances in NLP by incentivizing the creation of more accurate models. While this leaderboard paradigm has been remarkably successful, a historical focus on performance-based evaluation has been at the expense of other qualities that the NLP community values in models, such as compactness, fairness, and energy efficiency. In this opinion paper, we study the divergence between what is incentivized by leaderboards and what is useful in practice through the lens of microeconomic theory. We frame both the leaderboard and NLP practitioners as consumers and the benefit they get from a model as its utility to them. With this framing, we formalize how leaderboards -- in their current form -- can be poor proxies for the NLP community at large. For example, a highly inefficient model would provide less utility to practitioners but not to a leaderboard, since it is a cost that only the former must bear. To allow practitioners to better estimate a model's utility to them, we advocate for more transparency on leaderboards, such as the reporting of statistics that are of practical concern (e.g., model size, energy efficiency, and inference latency).