# Learning from Crowds with Crowd-Kit

arXiv: https://arxiv.org/abs/2109.08584 · HTML: https://mlsanity.com/paper/2109.08584/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Sep 2021.

## Built on by (most cited first, 24 of 24)

- [If in a Crowdsourced Data Annotation Pipeline, a GPT-4](https://mlsanity.com/paper/2402.16795/) (2024)
- [Best Prompts for Text-to-Image Models and How to Find Them](https://mlsanity.com/paper/2209.11711/) (2022)
- [RuBLiMP: Russian Benchmark of Linguistic Minimal Pairs](https://mlsanity.com/paper/2406.19232/) (2024)
- [SCoFT: Self-Contrastive Fine-Tuning for Equitable Image Generation](https://mlsanity.com/paper/2401.08053/) (2024)
- [Culture-TRIP: Culturally-Aware Text-to-Image Generation with Iterative Prompt Refinement](https://mlsanity.com/paper/2502.16902/) (2025)
- [Active Preference Learning for Ordering Items In- and Out-of-sample](https://mlsanity.com/paper/2405.03059/) (2024)
- [ARTICLE: Annotator Reliability Through In-Context Learning](https://mlsanity.com/paper/2409.12218/) (2024)
- [Aggregating Soft Labels from Crowd Annotations Improves Uncertainty Estimation Under Distribution Shift](https://mlsanity.com/paper/2212.09409/) (2022)
- [Reliable, Reproducible, and Really Fast Leaderboards with Evalica](https://mlsanity.com/paper/2412.11314/) (2024)
- [Crowd-Certain: Label Aggregation in Crowdsourced and Ensemble Learning Classification](https://mlsanity.com/paper/2310.16293/) (2023)
- [Towards Equitable Agile Research and Development of AI and Robotics](https://mlsanity.com/paper/2402.08242/) (2024)
- [Hands-On Tutorial: Labeling with LLM and Human-in-the-Loop](https://mlsanity.com/paper/2411.04637/) (2024)
- [Handwritten Text Recognition from Crowdsourced Annotations](https://mlsanity.com/paper/2306.10878/) (2023)
- [Efficient Annotator Reliability Assessment with EffiARA](https://mlsanity.com/paper/2504.00589/) (2025)
- [Extending a Parliamentary Corpus with MPs' Tweets: Automatic Annotation and Evaluation Using MultiParTweet](https://mlsanity.com/paper/2512.11567/) (2025)
- [Towards Long-term Annotators: A Supervised Label Aggregation Baseline](https://mlsanity.com/paper/2311.14709/) (2023)
- [A Dataset for the Validation of Truth Inference Algorithms Suitable for Online Deployment](https://mlsanity.com/paper/2403.08826/) (2024)
- [WISE: Weighted Iterative Society-of-Experts for Robust Multimodal Multi-Agent Debate](https://mlsanity.com/paper/2512.02405/) (2025)
- [OpenAutoNLU: Open Source AutoML Library for NLU](https://mlsanity.com/paper/2603.01824/) (2026)
- [A Model for Imbalanced Label Aggregation: A Focus on Minority-Class Detection](https://mlsanity.com/paper/2607.24622/) (2026)
- [ExeCRE: Execution-Consistency Guided Reliability Estimation for Self-Correcting Code Generation](https://mlsanity.com/paper/2608.04439/) (2026)
- [Experts Rise Where LLMs Disagree: Using Cross-Model Disagreement to Target Expert Effort in LLM Codebook Revision for Large-Scale Annotation](https://mlsanity.com/paper/2609.26926/) (2026)
- [Beyond Keywords: Leveraging Generative LLMs and Label Aggregation to Classify Economic Policy Uncertainty in News Articles](https://mlsanity.com/paper/2609.35856/) (2026)
- [On the Calibration of Crowdsourcing Aggregators](https://mlsanity.com/paper/2610.16815/) (2026)

## Abstract (arXiv)

This paper presents Crowd-Kit, a general-purpose computational quality control toolkit for crowdsourcing. Crowd-Kit provides efficient and convenient implementations of popular quality control algorithms in Python, including methods for truth inference, deep learning from crowds, and data quality estimation. Our toolkit supports multiple modalities of answers and provides dataset loaders and example notebooks for faster prototyping. We extensively evaluated our toolkit on several datasets of different natures, enabling benchmarking computational quality control methods in a uniform, systematic, and reproducible way using the same codebase. We release our code and data under the Apache License 2.0 at https://github.com/Toloka/crowd-kit.