# BotEval: Facilitating Interactive Human Evaluation

arXiv: https://arxiv.org/abs/2407.17770 · HTML: https://mlsanity.com/paper/2407.17770/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 24 references that are in the map.
- First posted on arXiv 25 Jul 2024.

## Builds on (most cited first, 13 of 13)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [ParlAI: A Dialog Research Software Platform](https://mlsanity.com/paper/1705.06476/) (2017)
- [ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons](https://mlsanity.com/paper/1909.03087/) (2019)
- [POTATO: The Portable Text Annotation Tool](https://mlsanity.com/paper/2212.08620/) (2022)
- [CaSiNo: A Corpus of Campsite Negotiation Dialogues for Automatic Negotiation Systems](https://mlsanity.com/paper/2103.15721/) (2021)
- [Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents](https://mlsanity.com/paper/2201.04723/) (2022)
- [RECAP: Retrieval-Enhanced Context-Aware Prefix Encoder for Personalized Dialogue Response Generation](https://mlsanity.com/paper/2306.07206/) (2023)
- [Grounding Conversations with Improvised Dialogues](https://mlsanity.com/paper/2004.09544/) (2020)
- [Reflect, Not Reflex: Inference-Based Common Ground Improves Dialogue Response Quality](https://mlsanity.com/paper/2211.09267/) (2022)
- [Can Language Model Moderators Improve the Health of Online Discourse?](https://mlsanity.com/paper/2311.10781/) (2023)
- [Mephisto: A Framework for Portable, Reproducible, and Iterative Crowdsourcing](https://mlsanity.com/paper/2301.05154/) (2023)
- [LIDA: Lightweight Interactive Dialogue Annotator](https://mlsanity.com/paper/1911.01599/) (2019)

## Built on by (most cited first, 1 of 1)

- [SkillAtlas: An Attack Trace Library for Agent Skills](https://mlsanity.com/paper/2609.13353/) (2026)

## Abstract (arXiv)

Following the rapid progress in natural language processing (NLP) models, language models are applied to increasingly more complex interactive tasks such as negotiations and conversation moderations. Having human evaluators directly interact with these NLP models is essential for adequately evaluating the performance on such interactive tasks. We develop BotEval, an easily customizable, open-source, evaluation toolkit that focuses on enabling human-bot interactions as part of the evaluation process, as opposed to human evaluators making judgements for a static input. BotEval balances flexibility for customization and user-friendliness by providing templates for common use cases that span various degrees of complexity and built-in compatibility with popular crowdsourcing platforms. We showcase the numerous useful features of BotEval through a study that evaluates the performance of various chatbots on their effectiveness for conversational moderation and discuss how BotEval differs from other annotation tools.