# Lessons Learned from a Citizen Science Project for Natural Language Processing

arXiv: https://arxiv.org/abs/2304.12836 · HTML: https://mlsanity.com/paper/2304.12836/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 56 references that are in the map.
- First posted on arXiv 25 Apr 2023.

## Builds on (most cited first, 5 of 5)

- [Seeing Things from a Different Angle: Discovering Diverse Perspectives about Claims](https://mlsanity.com/paper/1906.03538/) (2019)
- [Argumentation Mining in User-Generated Web Discourse](https://mlsanity.com/paper/1601.02403/) (2016)
- [Beyond Fair Pay: Ethical Implications of NLP Crowdsourcing](https://mlsanity.com/paper/2104.10097/) (2021)
- [Annotation Curricula to Implicitly Train Non-Expert Annotators](https://mlsanity.com/paper/2106.02382/) (2021)
- [Gamified Crowdsourcing for Idiom Corpora Construction](https://mlsanity.com/paper/2102.00881/) (2021)

## Built on by (most cited first, 4 of 4)

- [The Future of Open Human Feedback](https://mlsanity.com/paper/2408.16961/) (2024)
- [Wisdom of Instruction-Tuned Language Model Crowds. Exploring Model Label Variation](https://mlsanity.com/paper/2307.12973/) (2023)
- [EVENT5Ws: A Large Dataset for Open-Domain Event Extraction from Documents](https://mlsanity.com/paper/2604.21890/) (2026)
- [The Annotation Scarcity Paradox in Low-Resource NLP Evaluation: A Decade of Acceleration and Emerging Constraints](https://mlsanity.com/paper/2605.19066/) (2026)

## Abstract (arXiv)

Many Natural Language Processing (NLP) systems use annotated corpora for training and evaluation. However, labeled data is often costly to obtain and scaling annotation projects is difficult, which is why annotation tasks are often outsourced to paid crowdworkers. Citizen Science is an alternative to crowdsourcing that is relatively unexplored in the context of NLP. To investigate whether and how well Citizen Science can be applied in this setting, we conduct an exploratory study into engaging different groups of volunteers in Citizen Science for NLP by re-annotating parts of a pre-existing crowdsourced dataset. Our results show that this can yield high-quality annotations and attract motivated volunteers, but also requires considering factors such as scalability, participation over time, and legal and ethical issues. We summarize lessons learned in the form of guidelines and provide our code and data to aid future work on Citizen Science.