mlsanityData to 7 Oct 2026

Yes-Yes-Yes: Proactive Data Collection for ACL Rolling Review and Beyond

arXiv 2201.11443 ↗First posted 27 Jan 2022cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

The shift towards publicly available text sources has enabled language processing at unprecedented scale, yet leaves under-serviced the domains where public and openly licensed data is scarce. Proactively collecting text data for research is a viable strategy to address this scarcity, but lacks systematic methodology taking into account the many ethical, legal and confidentiality-related aspects of data collection. Our work presents a case study on proactive data collection in peer review -- a challenging and under-resourced NLP domain. We outline ethical and legal desiderata for proactive data collection and introduce "Yes-Yes-Yes", the first donation-based peer reviewing data collection workflow that meets these requirements. We report on the implementation of Yes-Yes-Yes at ACL Rolling Review and empirically study the implications of proactive data collection for the dataset size and the biases induced by the donation behavior on the peer reviewing platform.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 16 of its 33 references that are in the map. First posted on arXiv 27 Jan 2022.

Builds on

16 of 33 references.

References · 16

  1. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  2. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  3. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  4. 2018A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applicationscited by 135
  5. 2021Can We Automate Scientific Reviewing?cited by 63
  6. 2021Just What do You Think You're Doing, Dave?' A Checklist for Responsible Data Use in NLPcited by 47
  7. 2021Addressing "Documentation Debt" in Machine Learning Research: A Retrospective Datasheet for BookCorpuscited by 46
  8. 2019Argument Mining for Understanding Peer Reviewscited by 41
  9. 2016What to do about non-standard (or non-canonical) language in NLPcited by 35
  10. 2020What Can We Do to Improve Peer Review in NLP?cited by 32
  11. 2019Does My Rebuttal Matter? Insights from a Major NLP Conferencecited by 31
  12. 2020Catch Me if I Can: Detecting Strategic Behaviour in Peer Assessmentcited by 14
4 more
  1. 2017A Longitudinal Assessment of the Persistence of Twitter Datasetscited by 11
  2. 2021Translational NLP: A New Paradigm and General Principles for Natural Language Processing Researchcited by 7
  3. 2020A Large Scale Randomized Controlled Trial on Herding in Peer-Review Discussionscited by 6
  4. 2021Assisting Decision Making in Scholarly Peer Review: A Preference Learning Perspectivecited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 1; 2024: 2; 2025: 3; 2026: 4.

12 papers.

  1. 2024Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviewscited by 160
  2. 2022NLPeer: A Unified Resource for the Computational Study of Peer Reviewcited by 60
  3. 2024What Can Natural Language Processing Do for Peer Review?cited by 24
  4. 2025PeerQA: A Scientific Question Answering Dataset from Peer Reviewscited by 21
  5. 2023Automatic Analysis of Substantiation in Scientific Peer Reviewscited by 19
  6. 2025LazyReview A Dataset for Uncovering Lazy Thinking in NLP Peer Reviewscited by 13
  7. 2025Large Language Models Penetration in Scholarly Writing and Peer Reviewcited by 6
  8. 2026Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Futurecited by 6
  9. 2026Review Arcade: On the Human Alignment and Gameability of LLM Reviewscited by 1
  10. 2022An Inclusive Notion of Textcited by 0
  11. 2026Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Timecited by 0
  12. 2026Reviewing the Reviewer: LLM-Assisted Reviewer Feedback Generation for Guideline Compliancecited by 0