mlsanityData to 7 Oct 2026

How not to Lie with a Benchmark: Rearranging NLP Leaderboards

arXiv 2112.01342 ↗First posted 2 Dec 2021cs.CL cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

Comparison with a human is an essential requirement for a benchmark for it to be a reliable measurement of model capabilities. Nevertheless, the methods for model comparison could have a fundamental flaw - the arithmetic mean of separate metrics is used for all tasks of different complexity, different size of test and training sets. In this paper, we examine popular NLP benchmarks' overall scoring methods and rearrange the models by geometric and harmonic mean (appropriate for averaging rates) according to their reported results. We analyze several popular benchmarks including GLUE, SuperGLUE, XGLUE, and XTREME. The analysis shows that e.g. human level on SuperGLUE is still not reached, and there is still room for improvement for the current models.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 14 of its 23 references that are in the map. First posted on arXiv 2 Dec 2021.

Builds on

14 of 23 references.

References · 14

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  3. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  4. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  5. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  6. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  7. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  8. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 235
  9. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 221
  10. 2019Show Your Work: Improved Reporting of Experimental Resultscited by 164
  11. 2021The Benchmark Lotterycited by 133
  12. 2021Memorization vs. Generalization: Quantifying Data Leakage in NLP Performance Evaluationcited by 69
2 more
  1. 2019Human vs. Muppet: A Conservative Estimate of Human Performance on the GLUE Benchmarkcited by 32
  2. 2020Utility is in the Eye of the User: A Critique of NLP Leaderboardscited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 1; 2024: 1; 2025: 2; 2026: 1.

9 papers.

  1. 2023LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domaincited by 68
  2. 2022Processing Long Legal Documents with Pre-trained Transformers: Modding LegalBERT and Longformercited by 20
  3. 2022Vote'n'Rank: Revision of Benchmarking with Social Choice Theorycited by 15
  4. 2024ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended Capabilitiescited by 13
  5. 2022Slovene SuperGLUE Benchmark: Translation and Evaluationcited by 7
  6. 2022Automatic Rule Induction for Interpretable Semi-Supervised Learningcited by 6
  7. 2025Clone-Robust Weights in Metric Spaces: Handling Redundancy Bias for Benchmark Aggregationcited by 1
  8. 2025Beyond statistical significance: Quantifying uncertainty and statistical variability in multilingual and multitask NLP evaluationcited by 0
  9. 2026LLM Olympiad: Why Model Evaluation Needs a Sealed Examcited by 0