# A Modular Task-oriented Dialogue System Using a Neural Mixture-of-Experts

arXiv: https://arxiv.org/abs/1907.05346 · HTML: https://mlsanity.com/paper/1907.05346/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in MoE large language models, one of 12 directions in Mixture-of-experts.
- Builds on 23 of its 35 references that are in the map.
- First posted on arXiv 10 Jul 2019.

## Placed in

- Mixture-of-experts / [MoE large language models](https://mlsanity.com/moe/moe-large-language-models/)

## Builds on (most cited first, 23 of 23)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [A Network-based End-to-End Trainable Task-oriented Dialogue System](https://mlsanity.com/paper/1604.04562/) (2016)
- [A Persona-Based Neural Conversation Model](https://mlsanity.com/paper/1603.06155/) (2016)
- [Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models](https://mlsanity.com/paper/1507.04808/) (2015)
- [A Neural Conversational Model](https://mlsanity.com/paper/1506.05869/) (2015)
- [A Survey on Dialogue Systems: Recent Advances and New Frontiers](https://mlsanity.com/paper/1711.01731/) (2017)
- [Key-Value Retrieval Networks for Task-Oriented Dialogue](https://mlsanity.com/paper/1705.05414/) (2017)
- [Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems](https://mlsanity.com/paper/1508.01745/) (2015)
- [Learning End-to-End Goal-Oriented Dialog](https://mlsanity.com/paper/1605.07683/) (2016)
- [A Neural Network Approach to Context-Sensitive Generation of Conversational Responses](https://mlsanity.com/paper/1506.06714/) (2015)
- [Multi-Source Domain Adaptation with Mixture of Experts](https://mlsanity.com/paper/1809.02256/) (2018)
- [Global-Locally Self-Attentive Dialogue State Tracker](https://mlsanity.com/paper/1805.09655/) (2018)
- [Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators](https://mlsanity.com/paper/1904.13015/) (2019)
- [Hybrid Code Networks: practical and efficient end-to-end dialog control with supervised and reinforcement learning](https://mlsanity.com/paper/1702.03274/) (2017)
- [Multi-task learning for Joint Language Understanding and Dialogue State Tracking](https://mlsanity.com/paper/1811.05408/) (2018)
- [A Context-aware Natural Language Generator for Dialogue Systems](https://mlsanity.com/paper/1608.07076/) (2016)
- [Granger-causal Attentive Mixtures of Experts: Learning Important Features with Neural Networks](https://mlsanity.com/paper/1802.02195/) (2018)
- [Sequential Dialogue Context Modeling for Spoken Language Understanding](https://mlsanity.com/paper/1705.03455/) (2017)
- [Dynamic Time-Aware Attention to Speaker Roles and Contexts for Spoken Language Understanding](https://mlsanity.com/paper/1710.00165/) (2017)

## Built on by (most cited first, 4 of 4)

- [Pretraining the Noisy Channel Model for Task-Oriented Dialogue](https://mlsanity.com/paper/2103.10518/) (2021)
- [ReMeDi: Resources for Multi-domain, Multi-service, Medical Dialogues](https://mlsanity.com/paper/2109.00430/) (2021)
- [A Cooperative Memory Network for Personalized Task-oriented Dialogue Systems with Incomplete User Profiles](https://mlsanity.com/paper/2102.08322/) (2021)
- [Multi-Type Context-Aware Conversational Recommender Systems via Mixture-of-Experts](https://mlsanity.com/paper/2504.13655/) (2025)

## Abstract (arXiv)

End-to-end Task-oriented Dialogue Systems (TDSs) have attracted a lot of attention for their superiority (e.g., in terms of global optimization) over pipeline modularized TDSs. Previous studies on end-to-end TDSs use a single-module model to generate responses for complex dialogue contexts. However, no model consistently outperforms the others in all cases. We propose a neural Modular Task-oriented Dialogue System(MTDS) framework, in which a few expert bots are combined to generate the response for a given dialogue context. MTDS consists of a chair bot and several expert bots. Each expert bot is specialized for a particular situation, e.g., one domain, one type of action of a system, etc. The chair bot coordinates multiple expert bots and adaptively selects an expert bot to generate the appropriate response. We further propose a Token-level Mixture-of-Expert (TokenMoE) model to implement MTDS, where the expert bots predict multiple tokens at each timestamp and the chair bot determines the final generated token by fully taking into consideration the outputs of all expert bots. Both the chair bot and the expert bots are jointly trained in an end-to-end fashion. To verify the effectiveness of TokenMoE, we carry out extensive experiments on a benchmark dataset. Compared with the baseline using a single-module model, our TokenMoE improves the performance by 8.1% of inform rate and 0.8% of success rate.