mlsanityData to 7 Oct 2026

Dynabench: Rethinking Benchmarking in NLP

arXiv 2104.14337 ↗First posted 7 Apr 2021cs.CL cs.AIcited by 447

Not on the map. Why?

Cited by 24 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce Dynabench, an open-source platform for dynamic dataset creation and model benchmarking. Dynabench runs in a web browser and supports human-and-model-in-the-loop dataset creation: annotators seek to create examples that a target model will misclassify, but that another person will not. In this paper, we argue that Dynabench addresses a critical need in our community: contemporary models quickly achieve outstanding performance on benchmark tasks but nonetheless fail on simple challenge examples and falter in real-world scenarios. With Dynabench, dataset creation, model development, and model assessment can directly inform each other, leading to more robust and informative benchmarks. We report on four initial NLP tasks, illustrating these concepts and highlighting the promise of the platform, and address potential objections to dynamic benchmarking as a new standard for the field.

Key facts. Cited by 447 papers in the map (data through 7 Oct 2026). Builds on 74 of its 107 references that are in the map. First posted on arXiv 7 Apr 2021.

Builds on

74 of 107 references.

References · 74

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  9. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  10. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  12. 2020Learning to summarize from human feedbackcited by 2,562
38 more
  1. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  2. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  3. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  4. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  5. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  6. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  7. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  8. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  9. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  10. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  11. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  12. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  13. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  14. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  15. 2018Gender Bias in Coreference Resolutioncited by 438
  16. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
  17. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 423
  18. 2019On Measuring Social Biases in Sentence Encoderscited by 397
  19. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  20. 2017Synthetic and Natural Noise Both Break Neural Machine Translationcited by 304
  21. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  22. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 291
  23. 2016Bidirectional Attention Flow for Machine Comprehensioncited by 278
  24. 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 260
  25. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 235
  26. 2020Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detectioncited by 217
  27. 2020HateCheck: Functional Tests for Hate Speech Detection Modelscited by 215
  28. 2018Stress Test Evaluation for Natural Language Inferencecited by 211
  29. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  30. 2020With Little Power Comes Great Responsibilitycited by 138
  31. 2020Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasetscited by 137
  32. 2018Breaking NLI Systems with Sentences that Require Simple Lexical Inferencescited by 134
  33. 2020Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehensioncited by 131
  34. 2017Understanding Abuse: A Typology of Abusive Language Detection Subtaskscited by 115
  35. 2020How Can We Accelerate Progress Towards Human-like Linguistic Generalization?cited by 114
  36. 2019Learning from Dialogue after Deployment: Feed Yourself, Chatbot!cited by 102
  37. 2018On Adversarial Examples for Character-Level Neural Machine Translationcited by 85
  38. 2018On the Impact of Various Types of Noise on Neural Machine Translationcited by 76

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 29; 2022: 80; 2023: 68; 2024: 69; 2025: 81; 2026: 120.

447 papers.

Sort:

By area:

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  3. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  4. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  5. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  6. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  7. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  8. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  9. 2025Humanity's Last Examcited by 467
  10. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  11. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
  12. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
38 more
  1. 2023Challenges and Applications of Large Language Modelscited by 251
  2. 2022CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilitiescited by 245
  3. 2023Can Large Language Models Transform Computational Social Science?cited by 245
  4. 2021Sustainable AI: Environmental Implications, Challenges and Opportunitiescited by 242
  5. 2022RealTime QA: What's the Answer Right Now?cited by 239
  6. 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 227
  7. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 200
  8. 2022BlenderBot 3: a deployed conversational agent that continually learns to responsibly engagecited by 197
  9. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 189
  10. 2024SciCode: A Research Coding Benchmark Curated by Scientistscited by 172
  11. 2022"I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Datasetcited by 160
  12. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  13. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  14. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Gamescited by 141
  15. 2023Auditing large language models: a three-layered approachcited by 135
  16. 2021The Benchmark Lotterycited by 133
  17. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 121
  18. 2024WebCanvas: Benchmarking Web Agents in Online Environmentscited by 119
  19. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 113
  20. 2022CommonsenseQA 2.0: Exposing the Limits of AI through Gamificationcited by 108
  21. 2021Measure and Improve Robustness in NLP Models: A Surveycited by 107
  22. 2025Measuring what Matters: Construct Validity in Large Language Model Benchmarkscited by 102
  23. 2022ReCode: Robustness Evaluation of Code Generation Modelscited by 100
  24. 2023DyVal: Dynamic Evaluation of Large Language Models for Reasoning Taskscited by 100
  25. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  26. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samplescited by 94
  27. 2022Evaluating Human-Language Model Interactioncited by 90
  28. 2022DataPerf: Benchmarks for Data-Centric AI Developmentcited by 89
  29. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedbackcited by 86
  30. 2025Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 86
  31. 2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 84
  32. 2023Anchor Points: Benchmarking Models with Much Fewer Examplescited by 84
  33. 2021Improving Question Answering Model Robustness with Synthetic Adversarial Data Generationcited by 77
  34. 2021Anticipating Safety Issues in E2E Conversational AI: Framework and Toolingcited by 77
  35. 2024Benchmark Self-Evolving: A Multi-Agent Framework for Dynamic LLM Evaluationcited by 73
  36. 2021NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentationcited by 72
  37. 2022GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspectivecited by 72
  38. 2023Unleashing the potential of prompt engineering for large language modelscited by 72

50 most cited of 447 All 447 (JSON)