# DialCrowd 2.0: A Quality-Focused Dialog System Crowdsourcing Toolkit

arXiv: https://arxiv.org/abs/2207.12551 · HTML: https://mlsanity.com/paper/2207.12551/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 30 references that are in the map.
- First posted on arXiv 25 Jul 2022.

## Builds on (most cited first, 4 of 4)

- [ParlAI: A Dialog Research Software Platform](https://mlsanity.com/paper/1705.06476/) (2017)
- [ConvLab-2: An Open-Source Toolkit for Building, Evaluating, and Diagnosing Dialogue Systems](https://mlsanity.com/paper/2002.04793/) (2020)
- [A Survey of NLP-Related Crowdsourcing HITs: what works and what does not](https://mlsanity.com/paper/2111.05241/) (2021)
- [MEEP: An Open-Source Platform for Human-Human Dialog Collection and End-to-End Agent Training](https://mlsanity.com/paper/2010.04747/) (2020)

## Built on by (most cited first, 6 of 6)

- [GenTUS: Simulating User Behaviour and Language in Task-oriented Dialogues with Generative Transformers](https://mlsanity.com/paper/2208.10817/) (2022)
- [Infusing Emotions into Task-oriented Dialogue Systems: Understanding, Management, and Generation](https://mlsanity.com/paper/2408.02417/) (2024)
- [All Learning Has an Emotional Basis, So Does Task-Oriented Dialogue](https://mlsanity.com/paper/2507.01594/) (2025)
- [DIALIGHT: Lightweight Multilingual Development and Evaluation of Task-Oriented Dialogue Systems with Large Language Models](https://mlsanity.com/paper/2401.02208/) (2024)
- [The DialPort tools](https://mlsanity.com/paper/2208.10918/) (2022)
- [Dialogue Systems Engineering: A Survey and Future Directions](https://mlsanity.com/paper/2508.02279/) (2025)

## Abstract (arXiv)

Dialog system developers need high-quality data to train, fine-tune and assess their systems. They often use crowdsourcing for this since it provides large quantities of data from many workers. However, the data may not be of sufficiently good quality. This can be due to the way that the requester presents a task and how they interact with the workers. This paper introduces DialCrowd 2.0 to help requesters obtain higher quality data by, for example, presenting tasks more clearly and facilitating effective communication with workers. DialCrowd 2.0 guides developers in creating improved Human Intelligence Tasks (HITs) and is directly applicable to the workflows used currently by developers and researchers.