# ConvLab-2: An Open-Source Toolkit for Building, Evaluating, and Diagnosing Dialogue Systems

arXiv: https://arxiv.org/abs/2002.04793 · HTML: https://mlsanity.com/paper/2002.04793/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Feb 2020.

## Built on by (most cited first, 30 of 52)

- [ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL](https://mlsanity.com/paper/2402.19446/) (2024)
- [Shades of BLEU, Flavours of Success: The Case of MultiWOZ](https://mlsanity.com/paper/2106.05555/) (2021)
- [AuGPT: Auxiliary Tasks and Data Augmentation for End-To-End Dialogue with Pre-Trained Language Models](https://mlsanity.com/paper/2102.05126/) (2021)
- [Reliable LLM-based User Simulator for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2402.13374/) (2024)
- [DuetSim: Building User Simulator with Dual Large Language Models for Task-Oriented Dialogues](https://mlsanity.com/paper/2405.13028/) (2024)
- [Domain-independent User Simulation with Transformers for Task-oriented Dialogue Systems](https://mlsanity.com/paper/2106.08838/) (2021)
- [GenTUS: Simulating User Behaviour and Language in Task-oriented Dialogues with Generative Transformers](https://mlsanity.com/paper/2208.10817/) (2022)
- [ConvLab-3: A Flexible Dialogue System Toolkit Based on a Unified Data Format](https://mlsanity.com/paper/2211.17148/) (2022)
- [EmoWOZ: A Large-Scale Corpus and Labelling Scheme for Emotion Recognition in Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2109.04919/) (2021)
- [Fantastic Rewards and How to Tame Them: A Case Study on Reward Learning for Task-oriented Dialogue Systems](https://mlsanity.com/paper/2302.10342/) (2023)
- [In-Context Learning User Simulators for Task-Oriented Dialog Systems](https://mlsanity.com/paper/2306.00774/) (2023)
- [User Simulation with Large Language Models for Evaluating Task-Oriented Dialogue](https://mlsanity.com/paper/2309.13233/) (2023)
- [X-RiSAWOZ: High-Quality End-to-End Multilingual Dialogue Datasets and Few-shot Agents](https://mlsanity.com/paper/2306.17674/) (2023)
- [Dynamic Dialogue Policy for Continual Reinforcement Learning](https://mlsanity.com/paper/2204.05928/) (2022)
- [Cross-lingual Intermediate Fine-tuning improves Dialogue State Tracking](https://mlsanity.com/paper/2109.13620/) (2021)
- [Metaphorical User Simulators for Evaluating Task-oriented Dialogue Systems](https://mlsanity.com/paper/2204.00763/) (2022)
- [Uncertainty Measures in Neural Belief Tracking and the Effects on Dialogue Policy Performance](https://mlsanity.com/paper/2109.04349/) (2021)
- [DEUS: A Data-driven Approach to Estimate User Satisfaction in Multi-turn Dialogues](https://mlsanity.com/paper/2103.01287/) (2021)
- [End-to-End Conversational Search for Online Shopping with Utterance Transfer](https://mlsanity.com/paper/2109.05460/) (2021)
- [DialCrowd 2.0: A Quality-Focused Dialog System Crowdsourcing Toolkit](https://mlsanity.com/paper/2207.12551/) (2022)
- [A Generative User Simulator with GPT-based Architecture and Goal State Tracking for Reinforced Multi-Domain Dialog Systems](https://mlsanity.com/paper/2210.08692/) (2022)
- [Enhancing End-to-End Multi-Task Dialogue Systems: A Study on Intrinsic Motivation Reinforcement Learning Algorithms for Improved Training and Adaptability](https://mlsanity.com/paper/2401.18040/) (2024)
- [Integrating Pretrained Language Model for Dialogue Policy Learning](https://mlsanity.com/paper/2111.01398/) (2021)
- [Post-processing Networks: Method for Optimizing Pipeline Task-oriented Dialogue Systems using Reinforcement Learning](https://mlsanity.com/paper/2207.12185/) (2022)
- [Dialogue Term Extraction using Transfer Learning and Topological Data Analysis](https://mlsanity.com/paper/2208.10448/) (2022)
- [clem:todd: A Framework for the Systematic Benchmarking of LLM-Based Task-Oriented Dialogue System Realisations](https://mlsanity.com/paper/2505.05445/) (2025)
- [A Hybrid Task-Oriented Dialog System with Domain and Task Adaptive Pretraining](https://mlsanity.com/paper/2102.04506/) (2021)
- [High-Quality Diversification for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2106.00891/) (2021)
- ["Think Before You Speak": Improving Multi-Action Dialog Policy by Planning Single-Action Dialogs](https://mlsanity.com/paper/2204.11481/) (2022)
- [Dialogue Evaluation with Offline Reinforcement Learning](https://mlsanity.com/paper/2209.00876/) (2022)

## Abstract (arXiv)

We present ConvLab-2, an open-source toolkit that enables researchers to build task-oriented dialogue systems with state-of-the-art models, perform an end-to-end evaluation, and diagnose the weakness of systems. As the successor of ConvLab (Lee et al., 2019b), ConvLab-2 inherits ConvLab's framework but integrates more powerful dialogue models and supports more datasets. Besides, we have developed an analysis tool and an interactive tool to assist researchers in diagnosing dialogue systems. The analysis tool presents rich statistics and summarizes common mistakes from simulated dialogues, which facilitates error analysis and system improvement. The interactive tool provides a user interface that allows developers to diagnose an assembled dialogue system by interacting with the system and modifying the output of each system component.