# DIALIGHT: Lightweight Multilingual Development and Evaluation of Task-Oriented Dialogue Systems with Large Language Models

arXiv: https://arxiv.org/abs/2401.02208 · HTML: https://mlsanity.com/paper/2401.02208/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 51 references that are in the map.
- First posted on arXiv 4 Jan 2024.

## Builds on (most cited first, 30 of 33)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [A Network-based End-to-End Trainable Task-oriented Dialogue System](https://mlsanity.com/paper/1604.04562/) (2016)
- [OpenChat: Advancing Open-source Language Models with Mixed-Quality Data](https://mlsanity.com/paper/2309.11235/) (2023)
- [What Does it Mean for a Language Model to Preserve Privacy?](https://mlsanity.com/paper/2202.05520/) (2022)
- [Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset](https://mlsanity.com/paper/1909.05358/) (2019)
- [Unsupervised Evaluation of Interactive Dialog with DialoGPT](https://mlsanity.com/paper/2006.12719/) (2020)
- [Key-Value Retrieval Networks for Task-Oriented Dialogue](https://mlsanity.com/paper/1705.05414/) (2017)
- [Learning End-to-End Goal-Oriented Dialog](https://mlsanity.com/paper/1605.07683/) (2016)
- [MinTL: Minimalist Transfer Learning for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2009.12005/) (2020)
- [GALAXY: A Generative Pre-trained Model for Task-Oriented Dialog with Semi-Supervised Learning and Explicit Policy Injection](https://mlsanity.com/paper/2111.14592/) (2021)
- [A Comprehensive Assessment of Dialog Evaluation Metrics](https://mlsanity.com/paper/2106.03706/) (2021)
- [Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptation](https://mlsanity.com/paper/2305.15011/) (2023)
- [Multilingual Instruction Tuning With Just a Pinch of Multilinguality](https://mlsanity.com/paper/2401.01854/) (2024)
- [Are LLMs All You Need for Task-Oriented Dialogue?](https://mlsanity.com/paper/2304.06556/) (2023)
- [ConvLab-2: An Open-Source Toolkit for Building, Evaluating, and Diagnosing Dialogue Systems](https://mlsanity.com/paper/2002.04793/) (2020)
- [Shades of BLEU, Flavours of Success: The Case of MultiWOZ](https://mlsanity.com/paper/2106.05555/) (2021)
- [SGP-TOD: Building Task Bots Effortlessly via Schema-Guided LLM Prompting](https://mlsanity.com/paper/2305.09067/) (2023)
- [ChatGPT for Zero-shot Dialogue State Tracking: A Solution or an Opportunity?](https://mlsanity.com/paper/2306.01386/) (2023)
- [GlobalWoZ: Globalizing MultiWoZ to Develop Multilingual Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2110.07679/) (2021)
- [ConvLab: Multi-Domain End-to-End Dialog System Platform](https://mlsanity.com/paper/1904.08637/) (2019)
- [ConvLab-3: A Flexible Dialogue System Toolkit Based on a Unified Data Format](https://mlsanity.com/paper/2211.17148/) (2022)

## Built on by (most cited first, 1 of 1)

- [Leveraging Conversational Generative AI for Anomaly Detection in Digital Substations](https://mlsanity.com/paper/2411.16692/) (2024)

## Abstract (arXiv)

We present DIALIGHT, a toolkit for developing and evaluating multilingual Task-Oriented Dialogue (ToD) systems which facilitates systematic evaluations and comparisons between ToD systems using fine-tuning of Pretrained Language Models (PLMs) and those utilising the zero-shot and in-context learning capabilities of Large Language Models (LLMs). In addition to automatic evaluation, this toolkit features (i) a secure, user-friendly web interface for fine-grained human evaluation at both local utterance level and global dialogue level, and (ii) a microservice-based backend, improving efficiency and scalability. Our evaluations reveal that while PLM fine-tuning leads to higher accuracy and coherence, LLM-based systems excel in producing diverse and likeable responses. However, we also identify significant challenges of LLMs in adherence to task-specific instructions and generating outputs in multiple languages, highlighting areas for future research. We hope this open-sourced toolkit will serve as a valuable resource for researchers aiming to develop and properly evaluate multilingual ToD systems and will lower, currently still high, entry barriers in the field.