mlsanityData to 14 Oct 2026

What's the Meaning of Superhuman Performance in Today's NLU?

arXiv 2305.08414 ↗First posted 15 May 2023cs.CL cs.AIcited by 23

Not on the map. Why?

Abstract (arXiv)

In the last five years, there has been a significant focus in Natural Language Processing (NLP) on developing larger Pretrained Language Models (PLMs) and introducing benchmarks such as SuperGLUE and SQuAD to measure their abilities in language understanding, reasoning, and reading comprehension. These PLMs have achieved impressive results on these benchmarks, even surpassing human performance in some cases. This has led to claims of superhuman capabilities and the provocative idea that certain tasks have been solved. In this position paper, we take a critical look at these claims and ask whether PLMs truly have superhuman abilities and what the current benchmarks are really evaluating. We show that these benchmarks have serious limitations affecting the comparison between humans and PLMs and provide recommendations for fairer and more transparent benchmarks.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 51 of its 85 references that are in the map. First posted on arXiv 15 May 2023.

Builds on

51 of 85 references.

References · 51

  1. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  2. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  3. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  4. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  5. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  6. 2018Datasheets for Datasetscited by 1,386
  7. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  8. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  9. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  10. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  11. 2018CoQA: A Conversational Question Answering Challengecited by 683
  12. 2018Annotation Artifacts in Natural Language Inference Datacited by 671
38 more
  1. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  2. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
  3. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  4. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  5. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 425
  6. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 372
  7. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  8. 2021Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotationscited by 314
  9. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 295
  10. 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 264
  11. 2017Why We Need New Evaluation Metrics for NLGcited by 233
  12. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 222
  13. 2022Challenges and Strategies in Cross-Cultural NLPcited by 218
  14. 2019Probing Neural Network Comprehension of Natural Language Argumentscited by 214
  15. 2023ChatGPT: Jack of all trades, master of nonecited by 204
  16. 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 200
  17. 2019Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasetscited by 175
  18. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 168
  19. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 163
  20. 2021MultiVerS: Improving scientific claim verification with weak supervision and full-document contextcited by 143
  21. 2022The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluationcited by 139
  22. 2020Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasetscited by 138
  23. 2021On Releasing Annotator-Level Labels and Information in Datasetscited by 126
  24. 2022Jury Learning: Integrating Dissenting Voices into Machine Learning Modelscited by 118
  25. 2018How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarkscited by 88
  26. 2019Make Up Your Mind! Adversarial Generation of Inconsistent Natural Language Explanationscited by 71
  27. 2020We Need to Talk About Random Splitscited by 71
  28. 2019Garbage In, Garbage Out? Do Machine Learning Application Papers in Social Computing Report Where Human-Labeled Training Data Comes From?cited by 64
  29. 2020DynaSent: A Dynamic Benchmark for Sentiment Analysiscited by 63
  30. 2021Human-Adversarial Visual Question Answeringcited by 57
  31. 2022Don't Blame the Annotator: Bias Already Starts in the Annotation Instructionscited by 47
  32. 2022Annotation Error Detection: Analyzing the Past and Present for a More Coherent Futurecited by 45
  33. 2020A Set of Recommendations for Assessing Human-Machine Parity in Language Translationcited by 34
  34. 2019Human vs. Muppet: A Conservative Estimate of Human Performance on the GLUE Benchmarkcited by 32
  35. 2020To Test Machine Comprehension, Start by Defining Comprehensioncited by 31
  36. 2022Unobserved Local Structures Make Compositional Generalization Hardcited by 29
  37. 2022Investigating Reasons for Disagreement in Natural Language Inferencecited by 29
  38. 2022Structural generalization is hard for sequence-to-sequence modelscited by 24

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 9; 2025: 9; 2026: 2.

23 papers.

Sort:

By area:

  1. 2023Active Learning Principles for In-Context Learning with Large Language Modelscited by 54
  2. 2025Right Answer, Wrong Score: Uncovering the Inconsistencies of LLM Evaluation in Multiple-Choice Question Answeringcited by 28
  3. 2023BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language Modelscited by 18
  4. 2024Predictions from language models for multiple-choice tasks are not robust under variation of scoring methodscited by 13
  5. 2024The Call for Socially Aware Language Technologiescited by 13
  6. 2024mCSQA: Multilingual Commonsense Reasoning Dataset with Unified Creation Strategy by Language Models and Humanscited by 10
  7. 2025Has Machine Translation Evaluation Achieved Human Parity? The Human Reference and the Limits of Progresscited by 10
  8. 2025STREAM (ChemBio): A Standard for Transparently Reporting Evaluations in AI Model Reportscited by 10
  9. 2025Do Large Language Models Understand Word Senses?cited by 10
  10. 2024Natural Language Processing RELIES on Linguisticscited by 9
  11. 2024Is your benchmark truly adversarial? AdvScore: Evaluating Human-Grounded Adversarialnesscited by 7
  12. 2024On the Evaluation Practices in Multilingual NLP: Can Machine Translation Offer an Alternative to Human Translations?cited by 5
11 more
  1. 2023Establishing Trustworthiness: Rethinking Tasks and Model Evaluationcited by 4
  2. 2025EvalCards: A Framework for Standardized Evaluation Reportingcited by 4
  3. 2024BQA: Body Language Question Answering Dataset for Video Large Language Modelscited by 3
  4. 2025Text Classification Under Class Distribution Shift: A Surveycited by 2
  5. 2025Pay Attention to Real World Perturbations! Natural Robustness Evaluation in Machine Reading Comprehensioncited by 2
  6. 2024How the Advent of Ubiquitous Large Language Models both Stymie and Turbocharge Dynamic Adversarial Question Generationcited by 1
  7. 2024PATCH! {P}sychometrics-{A}ssis{T}ed Ben{CH}marking of Large Language Models against Human Populations: A Case Study of Proficiency in 8th Grade Mathematicscited by 1
  8. 2025A linguistically-motivated evaluation methodology for unraveling model's abilities in reading comprehension taskscited by 0
  9. 2025Designing LLMs for cultural sensitivity: Evidence from English-Japanese translationcited by 0
  10. 2026From annotation to reasoning: Culture in language modelscited by 0
  11. 2026NEPSEE: A Double-Annotated Benchmark for Nepali Sentiment Analysiscited by 0