# SOLOIST: Building Task Bots at Scale with Transfer Learning and Machine Teaching

arXiv: https://arxiv.org/abs/2005.05298 · HTML: https://mlsanity.com/paper/2005.05298/

## Key facts

- Cited by 71 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 May 2020.

## Built on by (most cited first, 30 of 71)

- [Structured Prediction as Translation between Augmented Natural Languages](https://mlsanity.com/paper/2101.05779/) (2021)
- [Dialogue State Tracking with a Language Model using Schema-Driven Prompting](https://mlsanity.com/paper/2109.07506/) (2021)
- [SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversations](https://mlsanity.com/paper/2104.08667/) (2021)
- [BiToD: A Bilingual Multi-Domain Dataset For Task-Oriented Dialogue Modeling](https://mlsanity.com/paper/2106.02787/) (2021)
- [Zero-Shot Dialogue State Tracking via Cross-Task Transfer](https://mlsanity.com/paper/2109.04655/) (2021)
- [SurgicalGPT: End-to-End Language-Vision GPT for Visual Question Answering in Surgery](https://mlsanity.com/paper/2304.09974/) (2023)
- [In-Context Learning for Few-Shot Dialogue State Tracking](https://mlsanity.com/paper/2203.08568/) (2022)
- [Neural Approaches to Conversational Information Retrieval](https://mlsanity.com/paper/2201.05176/) (2022)
- [Transferable Dialogue Systems and User Simulators](https://mlsanity.com/paper/2107.11904/) (2021)
- [CHAI: A CHatbot AI for Task-Oriented Dialogue with Offline Reinforcement Learning](https://mlsanity.com/paper/2204.08426/) (2022)
- [Large Language Models as Zero-shot Dialogue State Tracker through Function Calling](https://mlsanity.com/paper/2402.10466/) (2024)
- [Zero-shot Generalization in Dialog State Tracking through Generative Question Answering](https://mlsanity.com/paper/2101.08333/) (2021)
- [Knowledge-Grounded Dialogue Generation with a Unified Knowledge Representation](https://mlsanity.com/paper/2112.07924/) (2021)
- [Annotation Inconsistency and Entity Bias in MultiWOZ](https://mlsanity.com/paper/2105.14150/) (2021)
- [CINS: Comprehensive Instruction for Few-shot Learning in Task-oriented Dialog Systems](https://mlsanity.com/paper/2109.04645/) (2021)
- [Effectiveness of Pre-training for Few-shot Intent Classification](https://mlsanity.com/paper/2109.05782/) (2021)
- [Pretraining the Noisy Channel Model for Task-Oriented Dialogue](https://mlsanity.com/paper/2103.10518/) (2021)
- [RetGen: A Joint framework for Retrieval and Grounded Text Generation Modeling](https://mlsanity.com/paper/2105.06597/) (2021)
- [Context-Aware Language Modeling for Goal-Oriented Dialogue Systems](https://mlsanity.com/paper/2204.10198/) (2022)
- [Dialogue Summaries as Dialogue States (DS2), Template-Guided Summarization for Few-shot Dialogue State Tracking](https://mlsanity.com/paper/2203.01552/) (2022)
- [User Simulation with Large Language Models for Evaluating Task-Oriented Dialogue](https://mlsanity.com/paper/2309.13233/) (2023)
- [OrchestraLLM: Efficient Orchestration of Language Models for Dialogue State Tracking](https://mlsanity.com/paper/2311.09758/) (2023)
- [Self-training Improves Pre-training for Few-shot Learning in Task-oriented Dialog Systems](https://mlsanity.com/paper/2108.12589/) (2021)
- [LAD: Language Models as Data for Zero-Shot Dialog](https://mlsanity.com/paper/2207.14393/) (2022)
- [Domain State Tracking for a Simplified Dialogue System](https://mlsanity.com/paper/2103.06648/) (2021)
- [Learning from Language Description: Low-shot Named Entity Recognition via Decomposed Framework](https://mlsanity.com/paper/2109.05357/) (2021)
- [Towards Socially Intelligent Agents with Mental State Transition and Human Utility](https://mlsanity.com/paper/2103.07011/) (2021)
- [Task-Oriented Dialogue System as Natural Language Generation](https://mlsanity.com/paper/2108.13679/) (2021)
- [A Slot Is Not Built in One Utterance: Spoken Language Dialogs with Sub-Slots](https://mlsanity.com/paper/2203.10759/) (2022)
- [Surgical-LLaVA: Toward Surgical Scenario Understanding via Large Language and Vision Models](https://mlsanity.com/paper/2410.09750/) (2024)

## Abstract (arXiv)

We present a new method SOLOIST that uses transfer learning and machine teaching to build task bots at scale. We parameterize classical modular task-oriented dialog systems using a Transformer-based auto-regressive language model, which subsumes different dialog modules into a single neural model. We pre-train, on heterogeneous dialog corpora, a task-grounded response generation model, which can generate dialog responses grounded in user goals and real-world knowledge for task completion. The pre-trained model can be efficiently adapted to accomplish new tasks with a handful of task-specific dialogs via machine teaching, where training samples are generated by human teachers interacting with the system. Experiments show that (i) SOLOIST creates new state-of-the-art on well-studied task-oriented dialog benchmarks, including CamRest676 and MultiWOZ; (ii) in the few-shot fine-tuning settings, SOLOIST significantly outperforms existing methods, and (iii) the use of machine teaching substantially reduces the labeling cost of fine-tuning. The pre-trained models and codes are available at https://aka.ms/soloist.