mlsanityData to 7 Oct 2026

Targeting the Benchmark: On Methodology in Current Natural Language Processing Research

arXiv 2007.04792 ↗First posted 7 Jul 2020cs.CLcited by 44

Not on the map. Why?

Abstract (arXiv)

It has become a common pattern in our field: One group introduces a language task, exemplified by a dataset, which they argue is challenging enough to serve as a benchmark. They also provide a baseline model for it, which then soon is improved upon by other groups. Often, research efforts then move on, and the pattern repeats itself. What is typically left implicit is the argumentation for why this constitutes progress, and progress towards what. In this paper, we try to step back for a moment from this pattern and work out possible argumentations and their parts.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 10; 2023: 13; 2024: 5; 2025: 9; 2026: 3.

44 papers.

Sort:

By area:

  1. 2023PaLM 2 Technical Reportcited by 1,154
  2. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 340
  3. 2022The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluationcited by 133
  4. 2021Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Researchcited by 100
  5. 2023Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmarkcited by 62
  6. 2025Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluationcited by 57
  7. 2022Evaluation Gaps in Machine Learning Practicecited by 48
  8. 2023Benchmarks for Automated Commonsense Reasoning: A Surveycited by 40
  9. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 37
  10. 2022Underspecification in Scene Description-to-Depiction Taskscited by 30
  11. 2023It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performancecited by 25
  12. 2023Position: Key Claims in LLM Research Have a Long Tail of Footnotescited by 20
32 more
  1. 2023LeBenchmark 2.0: a Standardized, Replicable and Enhanced Framework for Self-supervised Representations of French Speechcited by 17
  2. 2022Dynatask: A Framework for Creating Dynamic AI Benchmark Taskscited by 16
  3. 2023Dialogue Games for Benchmarking Language Understanding: Motivation, Taxonomy, Strategycited by 12
  4. 2025The Benchmarking Epistemology: Validity Theory for Evaluating Machine Learning Modelscited by 12
  5. 2022Metaethical Perspectives on 'Benchmarking' AI Ethicscited by 11
  6. 2024Benchmark Data Repositories for Better Benchmarkingcited by 10
  7. 2021Is Stance Detection Topic-Independent and Cross-topic Generalizable? -- A Reproduction Studycited by 9
  8. 2022Experimental Standards for Deep Learning in Natural Language Processing Researchcited by 9
  9. 2023Evaluating Task-oriented Dialogue Systems: A Systematic Review of Measures, Constructs and their Operationalisationscited by 9
  10. 2022Language technology practitioners as language managers: arbitrating data bias and predictive bias in ASRcited by 7
  11. 2024Harmful YouTube Video Detection: A Taxonomy of Online Harm and MLLMs as Alternative Annotatorscited by 7
  12. 2025Benchmarking is Broken -- Don't Let AI be its Own Judgecited by 7
  13. 2026Computational Hermeneutics: Evaluating generative AI as a cultural technologycited by 7
  14. 2025Preliminary suggestions for rigorous GPAI model evaluationscited by 5
  15. 2023Establishing Trustworthiness: Rethinking Tasks and Model Evaluationcited by 4
  16. 2024Taking Action Towards Graceful Interaction: The Effects of Performing Actions on Modelling Policies for Instruction Clarification Requestscited by 4
  17. 2024On Tables with Numbers, with Numberscited by 4
  18. 2025What Should LLMs Forget? Quantifying Personal Data in LLMs for Right-to-Be-Forgotten Requestscited by 4
  19. 2021LoNLI: An Extensible Framework for Testing Diverse Logical Reasoning Capabilities for NLIcited by 3
  20. 2022StyLEx: Explaining Style Using Human Lexical Annotationscited by 3
  21. 2025Research Community Perspectives on "Intelligence" and Large Language Modelscited by 2
  22. 2022Fighting FIRe with FIRE: Assessing the Validity of Text-to-Video Retrieval Benchmarkscited by 1
  23. 2023Pento-DIARef: A Diagnostic Dataset for Learning the Incremental Algorithm for Referring Expression Generation from Examplescited by 1
  24. 2023On Degrees of Freedom in Defining and Testing Natural Language Understandingcited by 1
  25. 2023Explaining Relation Classification Models with Semantic Extentscited by 1
  26. 2026Human-Centred LLM Privacy Audits: Findings and Frictionscited by 1
  27. 2022An Interdisciplinary Perspective on Evaluation and Experimental Design for Visual Text Analytics: Position Papercited by 0
  28. 2023Weisfeiler and Leman Go Measurement Modeling: Probing the Validity of the WL Testcited by 0
  29. 2024LLMs as Function Approximators: Terminology, Taxonomy, and Questions for Evaluationcited by 0
  30. 2025A Conceptual Framework for AI Capability Evaluationscited by 0
  31. 2025Surprisal reveals diversity gaps in image captioning and different scorers change the storycited by 0
  32. 2026Accuracy Without Grounding: Diagnosing Visual Dependency Dissociation in Video LLM Benchmarkscited by 0