# Metritocracy: Representative Metrics for Lite Benchmarks

arXiv: https://arxiv.org/abs/2506.09813 · HTML: https://mlsanity.com/paper/2506.09813/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 32 references that are in the map.
- First posted on arXiv 11 Jun 2025.

## Builds on (most cited first, 16 of 16)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [tinyBenchmarks: evaluating LLMs with fewer examples](https://mlsanity.com/paper/2402.14992/) (2024)
- [Efficient Benchmarking of Language Models](https://mlsanity.com/paper/2308.11696/) (2023)
- [Proportionally Fair Clustering](https://mlsanity.com/paper/1905.03674/) (2019)
- [Justified Representation in Approval-Based Committee Voting](https://mlsanity.com/paper/1407.8269/) (2014)
- [Proportional Participatory Budgeting with Additive Utilities](https://mlsanity.com/paper/2008.13276/) (2020)
- [Active Evaluation Acquisition for Efficient LLM Benchmarking](https://mlsanity.com/paper/2410.05952/) (2024)
- [InfoLM: A New Metric to Evaluate Summarization & Data2Text Generation](https://mlsanity.com/paper/2112.01589/) (2021)
- [Inherent Trade-Offs between Diversity and Stability in Multi-Task Benchmarks](https://mlsanity.com/paper/2405.01719/) (2024)
- [Proportional Justified Representation](https://mlsanity.com/paper/1611.09928/) (2016)
- [How Robust are Model Rankings: A Leaderboard Customization Approach for Equitable Evaluation](https://mlsanity.com/paper/2106.05532/) (2021)
- [Vote'n'Rank: Revision of Benchmarking with Social Choice Theory](https://mlsanity.com/paper/2210.05769/) (2022)
- [Towards More Robust NLP System Evaluation: Handling Missing Scores in Benchmarks](https://mlsanity.com/paper/2305.10284/) (2023)
- [Proportional Fairness in Non-Centroid Clustering](https://mlsanity.com/paper/2410.23273/) (2024)
- [Full Proportional Justified Representation](https://mlsanity.com/paper/2501.12015/) (2025)

## Built on by (most cited first, 2 of 2)

- [Finding Common Ground in a Sea of Alternatives](https://mlsanity.com/paper/2603.16751/) (2026)
- [Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation](https://mlsanity.com/paper/2609.04298/) (2026)

## Abstract (arXiv)

A common problem in LLM evaluation is how to choose a subset of metrics from a full suite of possible metrics. Subset selection is usually done for efficiency or interpretability reasons, and the goal is often to select a ``representative'' subset of metrics. However, ``representative'' is rarely clearly defined. In this work, we use ideas from social choice theory to formalize two notions of representation for the selection of a subset of evaluation metrics. We first introduce positional representation, which guarantees every alternative is sufficiently represented at every position cutoff. We then introduce positional proportionality, which guarantees no alternative is proportionally over- or under-represented by more than a small error at any position. We prove upper and lower bounds on the smallest number of metrics needed to guarantee either of these properties in the worst case. We also study a generalized form of each property that allows for additional input on groups of metrics that must be represented. Finally, we tie theory to practice through real-world case studies on both LLM evaluation and hospital quality evaluation.