mlsanityData to 14 Oct 2026

Changing the World by Changing the Data

arXiv 2105.13947 ↗First posted 28 May 2021cs.CL cs.AIcited by 51

Not on the map. Why?

Abstract (arXiv)

NLP community is currently investing a lot more research and resources into development of deep learning models than training data. While we have made a lot of progress, it is now clear that our models learn all kinds of spurious patterns, social biases, and annotation artifacts. Algorithmic solutions have so far had limited success. An alternative that is being actively discussed is more careful design of datasets so as to deliver specific signals. This position paper maps out the arguments for and against data curation, and argues that fundamentally the point is moot: curation already is and will be happening, and it is changing the world. The question is only how much thought we want to invest into that process.

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). Builds on 37 of its 76 references that are in the map. First posted on arXiv 28 May 2021.

Builds on

37 of 76 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  5. 2020Extracting Training Data from Large Language Modelscited by 2,189
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  7. 2018Datasheets for Datasetscited by 1,386
  8. 2018Model Cards for Model Reportingcited by 1,080
  9. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  10. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
  11. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  12. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
25 more
  1. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  2. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  3. 2018Annotation Artifacts in Natural Language Inference Datacited by 671
  4. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 571
  5. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 541
  6. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 337
  7. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 320
  8. 2019Mitigating Gender Bias in Natural Language Processing: Literature Reviewcited by 287
  9. 2020Data and its (dis)contents: A survey of dataset development and use in machine learning researchcited by 260
  10. 2019Racial Bias in Hate Speech and Abusive Language Detection Datasetscited by 209
  11. 2020Social Biases in NLP Models as Barriers for Persons with Disabilitiescited by 182
  12. 2019Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasetscited by 175
  13. 2020Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasetscited by 138
  14. 2020Adversarial Watermarking Transformer: Towards Tracing Text Provenance with Data Hidingcited by 130
  15. 2019Lessons from Archives: Strategies for Collecting Sociocultural Data in Machine Learningcited by 116
  16. 2020How Can We Accelerate Progress Towards Human-like Linguistic Generalization?cited by 114
  17. 2020When Do You Need Billions of Words of Pretraining Data?cited by 104
  18. 2021Re-imagining Algorithmic Fairness in India and Beyondcited by 104
  19. 2021Competency Problems: On Finding and Removing Artifacts in Language Datacited by 91
  20. 2021Addressing "Documentation Debt" in Machine Learning Research: A Retrospective Datasheet for BookCorpuscited by 47
  21. 2020What Can We Do to Improve Peer Review in NLP?cited by 32
  22. 2020Universal linguistic inductive biases via meta-learningcited by 29
  23. 2019Posing Fair Generalization Tasks for Natural Language Inferencecited by 22
  24. 2020An Analysis of Dataset Overlap on Winograd-Style Taskscited by 16
  25. 2021The Slodderwetenschap (Sloppy Science) of Stochastic Parrots -- A Plea for Science to NOT take the Route Advocated by Gebru and Bendercited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 9; 2022: 17; 2023: 11; 2024: 7; 2025: 5; 2026: 2.

51 papers.

Sort:

By area:

  1. 2022Is GPT-3 a Good Data Annotator?cited by 208
  2. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 182
  3. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 155
  4. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  5. 2022Jury Learning: Integrating Dissenting Voices into Machine Learning Modelscited by 118
  6. 2023NLPositionality: Characterizing Design Biases of Datasets and Modelscited by 94
  7. 2021QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehensioncited by 91
  8. 2023The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AIcited by 73
  9. 2021Swiss-Judgment-Prediction: A Multilingual Legal Judgment Prediction Benchmarkcited by 60
  10. 2023Opening up ChatGPT: Tracking openness, transparency, and accountability in instruction-tuned text generatorscited by 56
  11. 2021Just What do You Think You're Doing, Dave?' A Checklist for Responsible Data Use in NLPcited by 47
  12. 2024Consent in Crisis: The Rapid Decline of the AI Data Commonscited by 46
38 more
  1. 2021The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Failcited by 38
  2. 2022BigBIO: A Framework for Data-Centric Biomedical Natural Language Processingcited by 36
  3. 2022Data Governance in the Age of Large-Scale Data-Driven Language Technologycited by 33
  4. 2023The ROOTS Search Tool: Data Transparency for LLMscited by 31
  5. 2022On the Impossible Safety of Large AI Modelscited by 28
  6. 2024Navigating LLM Ethics: Advancements, Challenges, and Future Directionscited by 28
  7. 2022Neural Pipeline for Zero-Shot Data-to-Text Generationcited by 19
  8. 2022On the Limitations of Dataset Balancing: The Lost Battle Against Spurious Correlationscited by 19
  9. 2021SynthBio: A Case Study in Human-AI Collaborative Curation of Text Datasetscited by 18
  10. 2023Reproducibility in NLP: What Have We Learned from the Checklist?cited by 18
  11. 2024Data Authenticity, Consent, & Provenance for AI are all broken: what will it take to fix them?cited by 15
  12. 2022Pre-training Data Quality and Quantity for a Low-Resource Language: New Corpus and BERT Models for Maltesecited by 14
  13. 2024Data, Data Everywhere: A Guide for Pretraining Dataset Constructioncited by 11
  14. 2023The timing bottleneck: Why timing and overlap are mission-critical for conversational user interfaces, speech recognition and dialogue systemscited by 9
  15. 2023Beyond Testers' Biases: Guiding Model Testing with Knowledge Bases using LLMscited by 9
  16. 2021Annotation Curricula to Implicitly Train Non-Expert Annotatorscited by 8
  17. 2022Why only Micro-F1? Class Weighting of Measures for Relation Classificationcited by 8
  18. 2021Dyna-bAbI: unlocking bAbI's potential with dynamic synthetic benchmarkingcited by 7
  19. 2024BLIP: Facilitating the Exploration of Undesirable Consequences of Digital Technologiescited by 6
  20. 2021DAG Card is the new Model Cardcited by 5
  21. 2022When Chosen Wisely, More Data Is What You Need: A Universal Sample-Efficient Strategy For Data Augmentationcited by 5
  22. 2023The Role of Data Curation in Image Captioningcited by 4
  23. 2023To Build Our Future, We Must Know Our Past: Contextualizing Paradigm Shifts in Natural Language Processingcited by 4
  24. 2022Back to the Future: On Potential Histories in NLPcited by 3
  25. 2022Building and curating conversational corpora for diversity-aware language science and technologycited by 2
  26. 2025We Need to Measure Data Diversity in NLP -- Better and Broadercited by 2
  27. 2025When Fairness Isn't Statistical: The Limits of Machine Learning in Evaluating Legal Reasoningcited by 2
  28. 2026The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotationcited by 2
  29. 2022Fighting FIRe with FIRE: Assessing the Validity of Text-to-Video Retrieval Benchmarkscited by 1
  30. 2022A Major Obstacle for NLP Research: Let's Talk about Time Allocation!cited by 1
  31. 2024Speciesism in Natural Language Processing Researchcited by 1
  32. 2025Highlighting Case Studies in LLM Literature Review of Interdisciplinary System Sciencecited by 1
  33. 2021No News is Good News: A Critique of the One Billion Word Benchmarkcited by 0
  34. 2022Lutma: a Frame-Making Tool for Collaborative FrameNet Developmentcited by 0
  35. 2022Towards Lexical Gender Inference: A Scalable Methodology using Online Databasescited by 0
  36. 2024AI Research is not Magic, it has to be Reproducible and Responsible: Challenges in the AI field from the Perspective of its PhD Studentscited by 0
  37. 2025Disaster Informatics after the COVID-19 Pandemic: Bibliometric and Topic Analysis based on Large-scale Academic Literaturecited by 0
  38. 2025Practising responsibility: Ethics in NLP as a hands-on coursecited by 0

50 most cited of 51 All 51 (JSON)