mlsanityData to 7 Oct 2026

Just What do You Think You're Doing, Dave?' A Checklist for Responsible Data Use in NLP

arXiv 2109.06598 ↗First posted 14 Sep 2021cs.CL cs.AIcited by 47

Not on the map. Why?

Abstract (arXiv)

A key part of the NLP ethics movement is responsible use of data, but exactly what that means or how it can be best achieved remain unclear. This position paper discusses the core legal and ethical principles for collection and sharing of textual data, and the tensions between them. We propose a potential checklist for responsible data (re-)use that could both standardise the peer review of conference submissions, as well as enable a more in-depth view of published research across the community. Our proposal aims to contribute to the development of a consistent standard for data (re-)use, embraced across NLP conferences.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 16 of its 79 references that are in the map. First posted on arXiv 14 Sep 2021.

Builds on

16 of 79 references.

References · 16

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2020Extracting Training Data from Large Language Modelscited by 2,144
  5. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  6. 2020Data and its (dis)contents: A survey of dataset development and use in machine learning researchcited by 259
  7. 2019Show Your Work: Improved Reporting of Experimental Resultscited by 164
  8. 2019A Step Toward Quantifying Independently Reproducible Machine Learning Researchcited by 77
  9. 2021Beyond Fair Pay: Ethical Implications of NLP Crowdsourcingcited by 52
  10. 2021Changing the World by Changing the Datacited by 51
  11. 2021Addressing "Documentation Debt" in Machine Learning Research: A Retrospective Datasheet for BookCorpuscited by 46
  12. 2021Benchmarking Differential Privacy and Federated Learning for BERT Modelscited by 28
4 more
  1. 2019This Thing Called Fairness: Disciplinary Confusion Realizing a Value in Technologycited by 24
  2. 2019On the Legal Compatibility of Fairness Definitionscited by 19
  3. 2021No Intruder, no Validity: Evaluation Criteria for Privacy-Preserving Text Anonymizationcited by 13
  4. 2021Use of Formal Ethical Reviews in NLP Literature: Historical Trends and Current Practicescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 9; 2023: 11; 2024: 15; 2025: 3; 2026: 8.

47 papers.

Sort:

  1. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  2. 2022One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesiacited by 65
  3. 2022Handling and Presenting Harmful Text in NLP Researchcited by 43
  4. 2023Quantifying the Dialect Gap and its Correlates Across Languagescited by 33
  5. 2022Data Governance in the Age of Large-Scale Data-Driven Language Technologycited by 32
  6. 2024PEDANTS: Cheap but Effective and Interpretable Answer Equivalencecited by 31
  7. 2024AnthroScore: A Computational Linguistic Measure of Anthropomorphismcited by 27
  8. 2024ECBD: Evidence-Centered Benchmark Design for NLPcited by 27
  9. 2024What Can Natural Language Processing Do for Peer Review?cited by 24
  10. 2023The Intended Uses of Automated Fact-Checking Artefacts: Why, How and Whocited by 19
  11. 2023Reproducibility in NLP: What Have We Learned from the Checklist?cited by 18
  12. 2025Reality Check: A New Evaluation Ecosystem Is Necessary to Understand AI's Real World Effectscited by 18
35 more
  1. 2023Multi3WOZ: A Multilingual, Multi-Domain, Multi-Parallel Dataset for Training and Evaluating Culturally Adapted Task-Oriented Dialog Systemscited by 16
  2. 2024Data Authenticity, Consent, & Provenance for AI are all broken: what will it take to fix them?cited by 15
  3. 2023When Good and Reproducible Results are a Giant with Feet of Clay: The Importance of Software Quality in NLPcited by 14
  4. 2023REFORMS: Reporting Standards for Machine Learning Based Sciencecited by 13
  5. 2022Yes-Yes-Yes: Proactive Data Collection for ACL Rolling Review and Beyondcited by 12
  6. 2023Pre-registration for Predictive Modelingcited by 11
  7. 2022Resources for Turkish Natural Language Processing: A critical surveycited by 6
  8. 2024Copycats: the many lives of a publicly available medical imaging datasetcited by 5
  9. 2025A Fully Generative Motivational Interviewing Counsellor Chatbot for Moving Smokers Towards the Decision to Quitcited by 5
  10. 2023NLP Reproducibility For All: Understanding Experiences of Beginnerscited by 4
  11. 2024Enhancing Question Answering on Charts Through Effective Pre-training Taskscited by 4
  12. 2024Building Better: Avoiding Pitfalls in Developing Language Resources when Data is Scarcecited by 4
  13. 2023Thorny Roses: Investigating the Dual Use Dilemma in Natural Language Processingcited by 3
  14. 2023RAI4IoE: Responsible AI for Enabling the Internet of Energycited by 3
  15. 2021Building Legal Datasetscited by 2
  16. 2022Building and curating conversational corpora for diversity-aware language science and technologycited by 2
  17. 2023torchdistill Meets Hugging Face Libraries for Reproducible, Coding-Free Deep Learning Studies: A Case Study on NLPcited by 2
  18. 2025Not quite Sherlock Holmes: Language model predictions do not reliably differentiate impossible from improbable eventscited by 2
  19. 2022One Venue, Two Conferences: The Separation of Chinese and American Citation Networkscited by 1
  20. 2024Modeling the Sacred: Considerations when Using Religious Texts in Natural Language Processingcited by 1
  21. 2024ConfReady: A RAG based Assistant and Dataset for Conference Checklist Responsescited by 1
  22. 2024Enhancing Data Quality through Simple De-duplication: Navigating Responsible Computational Social Science Researchcited by 1
  23. 2024Ethics Whitepaper: Whitepaper on Ethical Research into Large Language Modelscited by 1
  24. 2026Assessing socio-economic climate impacts from text datacited by 1
  25. 2026Illusions of the Gold Standard: A Large-scale Analysis of Human Evaluation Protocols for Long-form Text Generationcited by 1
  26. 2022An Inclusive Notion of Textcited by 0
  27. 2024CFMatch: Aligning Automated Answer Equivalence Evaluation with Expert Judgments For Open-Domain Question Answeringcited by 0
  28. 2024Demarked: A Strategy for Enhanced Abusive Speech Moderation through Counterspeech, Detoxification, and Message Managementcited by 0
  29. 2024A quest through interconnected datasets: lessons from highly-cited ICASSP paperscited by 0
  30. 2026Improving Reproducibility in Evaluation through Multi-Level Annotator Modelingcited by 0
  31. 2026Dial HEALTHDIAL for Advice: A Multilingual and Multi-Parallel Spoken Dialogue Dataset for Knowledge-Grounded Information Seekingcited by 0
  32. 2026The Language-Energy Divide: Measuring Energy Costs of Multilingual LLM Inferencecited by 0
  33. 2026The Future of NLP may not be at NLP Conferences: Scholarly Migration Patterns in Natural Language Processingcited by 0
  34. 2026Is the ACL Responsible NLP Checklist a Box-Ticking Exercise? A Large-Scale Analysis of EMNLP 2025cited by 0
  35. 2026Beyond Good Intentions: When Does the Framing of Multilingual and Low-Resource NLP Research Become a Caricature?cited by 0