mlsanityData to 7 Oct 2026

LEGOEval: An Open-Source Toolkit for Dialogue System Evaluation via Crowdsourcing

arXiv 2105.01992 ↗First posted 5 May 2021cs.AI cs.HCcited by 11

Not on the map. Why?

Abstract (arXiv)

We present LEGOEval, an open-source toolkit that enables researchers to easily evaluate dialogue systems in a few lines of code using the online crowdsource platform, Amazon Mechanical Turk. Compared to existing toolkits, LEGOEval features a flexible task design by providing a Python API that maps to commonly used React.js interface components. Researchers can personalize their evaluation procedures easily with our built-in pages as if playing with LEGO blocks. Thus, LEGOEval provides a fast, consistent method for reproducing human evaluation results. Besides the flexible task design, LEGOEval also offers an easy API to review collected data.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 11 of its 29 references that are in the map. First posted on arXiv 5 May 2021.

Builds on

11 of 29 references.

References · 11

  1. 2020Recipes for building an open-domain chatbotcited by 593
  2. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  3. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  4. 2015Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Modelscited by 207
  5. 2017ParlAI: A Dialog Research Software Platformcited by 139
  6. 2019Survey on Evaluation Methods for Dialogue Systemscited by 126
  7. 2017Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responsescited by 98
  8. 2019ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisonscited by 83
  9. 2020Towards Unified Dialogue System Evaluation: A Comprehensive Analysis of Current Evaluation Protocolscited by 35
  10. 2018A Hierarchical Latent Structure for Variational Conversation Modelingcited by 17
  11. 2020Perception Score, A Learned Metric for Open-ended Text Generation Evaluationcited by 7

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 2; 2023: 4; 2024: 1; 2025: 2.

11 papers.

  1. 2023PLACES: Prompting Language Models for Social Conversation Synthesiscited by 72
  2. 2023Prompt-Based Monte-Carlo Tree Search for Goal-Oriented Dialogue Policy Planningcited by 60
  3. 2023Controllable Mixed-Initiative Dialogue Generation through Promptingcited by 32
  4. 2024Strength Lies in Differences! Improving Strategy Planning for Non-collaborative Dialogues via Diversified User Simulationcited by 21
  5. 2021The R-U-A-Robot Dataset: Helping Avoid Chatbot Deception by Detecting User Questions About Human or Non-Human Identitycited by 16
  6. 2022Seamlessly Integrating Factual Information and Social Content with Persuasive Dialoguecited by 11
  7. 2025EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solvingcited by 10
  8. 2022Robots-Dont-Cry: Understanding Falsely Anthropomorphic Utterances in Dialog Systemscited by 8
  9. 2023Mephisto: A Framework for Portable, Reproducible, and Iterative Crowdsourcingcited by 7
  10. 2021ErAConD : Error Annotated Conversational Dialog Dataset for Grammatical Error Correctioncited by 2
  11. 2025Dialogue Systems Engineering: A Survey and Future Directionscited by 0