mlsanityData to 14 Oct 2026

Learning from Crowds with Crowd-Kit

arXiv 2109.08584 ↗First posted 17 Sep 2021cs.HC cs.SEcited by 24

Not on the map. Why?

Abstract (arXiv)

This paper presents Crowd-Kit, a general-purpose computational quality control toolkit for crowdsourcing. Crowd-Kit provides efficient and convenient implementations of popular quality control algorithms in Python, including methods for truth inference, deep learning from crowds, and data quality estimation. Our toolkit supports multiple modalities of answers and provides dataset loaders and example notebooks for faster prototyping. We extensively evaluated our toolkit on several datasets of different natures, enabling benchmarking computational quality control methods in a uniform, systematic, and reproducible way using the same codebase. We release our code and data under the Apache License 2.0 at https://github.com/Toloka/crowd-kit.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). First posted on arXiv 17 Sep 2021.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 3; 2024: 9; 2025: 4; 2026: 6.

24 papers.

Sort:

By area:

  1. 2024If in a Crowdsourced Data Annotation Pipeline, a GPT-4cited by 38
  2. 2022Best Prompts for Text-to-Image Models and How to Find Themcited by 35
  3. 2024RuBLiMP: Russian Benchmark of Linguistic Minimal Pairscited by 24
  4. 2024SCoFT: Self-Contrastive Fine-Tuning for Equitable Image Generationcited by 19
  5. 2025Culture-TRIP: Culturally-Aware Text-to-Image Generation with Iterative Prompt Refinementcited by 7
  6. 2024Active Preference Learning for Ordering Items In- and Out-of-samplecited by 6
  7. 2024ARTICLE: Annotator Reliability Through In-Context Learningcited by 6
  8. 2022Aggregating Soft Labels from Crowd Annotations Improves Uncertainty Estimation Under Distribution Shiftcited by 3
  9. 2024Reliable, Reproducible, and Really Fast Leaderboards with Evalicacited by 3
  10. 2023Crowd-Certain: Label Aggregation in Crowdsourced and Ensemble Learning Classificationcited by 2
  11. 2024Towards Equitable Agile Research and Development of AI and Roboticscited by 2
  12. 2024Hands-On Tutorial: Labeling with LLM and Human-in-the-Loopcited by 2
12 more
  1. 2023Handwritten Text Recognition from Crowdsourced Annotationscited by 1
  2. 2025Efficient Annotator Reliability Assessment with EffiARAcited by 1
  3. 2025Extending a Parliamentary Corpus with MPs' Tweets: Automatic Annotation and Evaluation Using MultiParTweetcited by 1
  4. 2023Towards Long-term Annotators: A Supervised Label Aggregation Baselinecited by 0
  5. 2024A Dataset for the Validation of Truth Inference Algorithms Suitable for Online Deploymentcited by 0
  6. 2025WISE: Weighted Iterative Society-of-Experts for Robust Multimodal Multi-Agent Debatecited by 0
  7. 2026OpenAutoNLU: Open Source AutoML Library for NLUcited by 0
  8. 2026A Model for Imbalanced Label Aggregation: A Focus on Minority-Class Detectioncited by 0
  9. 2026ExeCRE: Execution-Consistency Guided Reliability Estimation for Self-Correcting Code Generationcited by 0
  10. 2026Experts Rise Where LLMs Disagree: Using Cross-Model Disagreement to Target Expert Effort in LLM Codebook Revision for Large-Scale Annotationcited by 0
  11. 2026Beyond Keywords: Leveraging Generative LLMs and Label Aggregation to Classify Economic Policy Uncertainty in News Articlescited by 0
  12. 2026On the Calibration of Crowdsourcing Aggregatorscited by 0