mlsanityData to 7 Oct 2026

Comparing Test Sets with Item Response Theory

arXiv 2106.00840 ↗First posted 1 Jun 2021cs.CLcited by 55

Not on the map. Why?

Abstract (arXiv)

Recent years have seen numerous NLP datasets introduced to evaluate the performance of fine-tuned models on natural language understanding tasks. Recent results from large pretrained models, though, show that many of these datasets are largely saturated and unlikely to be able to detect further progress. What kind of datasets are still effective at discriminating among strong models, and what kind of datasets should we expect to be able to detect future improvements? To measure this uniformly across datasets, we draw on Item Response Theory and evaluate 29 datasets using predictions from 18 pretrained Transformer models on individual test examples. We find that Quoref, HellaSwag, and MC-TACO are best suited for distinguishing among state-of-the-art models, while SNLI, MNLI, and CommitmentBank seem to be saturated for current strong models. We also observe span selection task format, which is used for QA datasets like QAMR or SQuAD2.0, is effective in differentiating between strong and weak models.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Builds on 45 of its 68 references that are in the map. First posted on arXiv 1 Jun 2021.

Builds on

45 of 68 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  12. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
33 more
  1. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  2. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  3. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  4. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  5. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  6. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  7. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  8. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  9. 2018Pyro: Deep Universal Probabilistic Programmingcited by 379
  10. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  11. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  12. 2016NewsQA: A Machine Comprehension Datasetcited by 330
  13. 2013Black Box Variational Inferencecited by 317
  14. 2019Abductive Commonsense Reasoningcited by 300
  15. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 281
  16. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  17. 2019Probing Neural Network Comprehension of Natural Language Argumentscited by 212
  18. 2019MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehensioncited by 183
  19. 2020Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?cited by 127
  20. 2020Exploring and Predicting Transferability across NLP Taskscited by 126
  21. 2019"Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Commonsense Understandingcited by 121
  22. 2020MuTual: A Dataset for Multi-Turn Dialogue Reasoningcited by 114
  23. 2020When Do You Need Billions of Words of Pretraining Data?cited by 104
  24. 2019Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoningcited by 100
  25. 2018What Makes Reading Comprehension Questions Easier?cited by 59
  26. 2017The Argument Reasoning Comprehension Task: Identification and Reconstruction of Implicit Warrantscited by 55
  27. 2019Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowdscited by 40
  28. 2020jiant: A Software Toolkit for Research on General-Purpose Text Understanding Modelscited by 39
  29. 2019Human vs. Muppet: A Conservative Estimate of Human Performance on the GLUE Benchmarkcited by 32
  30. 2018MCScript: A Novel Dataset for Assessing Machine Comprehension Using Script Knowledgecited by 30
  31. 2017Crowdsourcing Question-Answer Meaning Representationscited by 21
  32. 2020Dynamic Data Selection for Curriculum Learning via Ability Estimationcited by 21
  33. 2020A Framework for Evaluation of Machine Reading Comprehension Gold Standardscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 5; 2023: 4; 2024: 12; 2025: 11; 2026: 20.

55 papers.

Sort:

By area:

  1. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 295
  2. 2021The Benchmark Lotterycited by 133
  3. 2024Efficient multi-prompt evaluation of LLMscited by 72
  4. 2024Quantifying Variance in Evaluation Benchmarkscited by 52
  5. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 51
  6. 2023Position: AI Evaluation Should Learn from How We Test Humanscited by 36
  7. 2021Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistantscited by 31
  8. 2025Fluid Language Model Benchmarkingcited by 30
  9. 2025Reliable and Efficient Amortized Model-based Evaluationcited by 29
  10. 2024Improving Model Evaluation using SMART Filtering of Benchmark Datasetscited by 20
  11. 2022py-irt: A Scalable Item Response Theory Library for Pythoncited by 19
  12. 2022ILDAE: Instance-Level Difficulty Analysis of Evaluation Datacited by 17
38 more
  1. 2024Label-Efficient Model Selection for Text Generationcited by 17
  2. 2024Raising the Bar: Investigating the Values of Large Language Models via Generative Evolving Testingcited by 17
  3. 2021Adversarially Constructed Evaluation Sets Are More Challenging, but May Not Be Faircited by 12
  4. 2025MERGE3: Efficient Evolutionary Merging on Consumer-grade GPUscited by 9
  5. 2025How Reliable is Language Model Micro-Benchmarking?cited by 8
  6. 2024Is your benchmark truly adversarial? AdvScore: Evaluating Human-Grounded Adversarialnesscited by 7
  7. 2025PredictaBoard: Benchmarking LLM Score Predictabilitycited by 7
  8. 2022Predicting Human Psychometric Properties Using Computational Language Modelscited by 5
  9. 2022On the data requirements of probingcited by 4
  10. 2024Compare without Despair: Reliable Preference Evaluation with Generation Separabilitycited by 4
  11. 2026Confident Rankings with Fewer Items: Adaptive LLM Evaluation with Continuous Scorescited by 4
  12. 2024TaskEval: Assessing Difficulty of Code Generation Tasks for Large Language Modelscited by 3
  13. 2025AutoJudger: An Agent-Driven Framework for Efficient Benchmarking of MLLMscited by 3
  14. 2026BenchMarker: An Education-Inspired Toolkit for Highlighting Flaws in Multiple-Choice Benchmarkscited by 3
  15. 2026Can We Trust Item Response Theory for AI Evaluation?cited by 3
  16. 2023Computational modeling of semantic changecited by 2
  17. 2023Towards Addressing the Misalignment of Object Proposal Evaluation for Vision-Language Tasks via Semantic Groundingcited by 2
  18. 2024The creative psychometric item generator: a framework for item generation and validation using large language modelscited by 2
  19. 2025LiveRAG: A diverse Q&A dataset with varying difficulty level for RAG evaluationcited by 2
  20. 2026ExpGuard: LLM Content Moderation in Specialized Domainscited by 2
  21. 2026AI Cartography: Mapping the Latent Landscape of AI Benchmark Ecosystemscited by 2
  22. 2026Auditing LLM Benchmarks with Item Response Theorycited by 2
  23. 2022MonoByte: A Pool of Monolingual Byte-level Language Modelscited by 1
  24. 2023On Degrees of Freedom in Defining and Testing Natural Language Understandingcited by 1
  25. 2024PATCH! {P}sychometrics-{A}ssis{T}ed Ben{CH}marking of Large Language Models against Human Populations: A Case Study of Proficiency in 8th Grade Mathematicscited by 1
  26. 2025LASER: Stratified Selective Sampling for Instruction Tuning with Dedicated Scoring Strategycited by 1
  27. 2025RIDE: Difficulty Evolving Perturbation with Item Response Theory for Mathematical Reasoningcited by 1
  28. 2026Cost-Efficient Estimation of General Abilities Across Benchmarkscited by 1
  29. 2026Offline Preference-Based Trajectory Evaluationcited by 1
  30. 2024On Evaluation of Vision Datasets and Models using Human Competency Frameworkscited by 0
  31. 2025A Conceptual Framework for AI Capability Evaluationscited by 0
  32. 2026Aligning Language Model Benchmarks with Pairwise Preferencescited by 0
  33. 2026Bayesian Inference and Decision Audits for Public Archives of Frontier AI Evaluationscited by 0
  34. 2026Dimensionality and Measurement Precision in HLE's Multiple-Choice Subsetcited by 0
  35. 2026Every Wrong Answer Counts: Option-Level Psychometrics for LLM Multiple-Choice Benchmarkscited by 0
  36. 2026NOLLI: A Difficulty-Calibrated Puzzle Benchmark for Diagnosing the English-Korean Performance Gapcited by 0
  37. 2026There Is No Neutral Harness: Modern LLM Leaderboards Are Manufactured by Config-Fragile Itemscited by 0
  38. 2026Rating the Raters: Rasch Measurement Theory for LLM Evaluationcited by 0

50 most cited of 55 All 55 (JSON)