mlsanityData to 14 Oct 2026

A Mixture-of-Expert Approach to RL-based Dialogue Management

arXiv 2206.00059 ↗First posted 31 May 2022cs.CL cs.AIcited by 16

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Despite recent advancements in language models (LMs), their application to dialogue management (DM) problems and ability to carry on rich conversations remain a challenge. We use reinforcement learning (RL) to develop a dialogue agent that avoids being short-sighted (outputting generic utterances) and maximizes overall user satisfaction. Most existing RL approaches to DM train the agent at the word-level, and thus, have to deal with a combinatorially complex action space even for a medium-size vocabulary. As a result, they struggle to produce a successful and engaging dialogue even if they are warm-started with a pre-trained LM. To address this issue, we develop a RL-based DM using a novel mixture of expert language model (MoE-LM) that consists of (i) a LM capable of learning diverse semantics for conversation histories, (ii) a number of em specialized LMs (or experts) capable of generating utterances corresponding to a particular attribute or personality, and (iii) a RL-based DM that performs dialogue planning with the utterances generated by the experts. Our MoE approach provides greater flexibility to generate sensible utterances with different intents and allows RL to focus on conversational-level DM. We compare it with SOTA baselines on open-domain dialogues and demonstrate its effectiveness both in terms of the diversity and sensibility of the generated utterances and the overall DM performance.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 47 of its 71 references that are in the map. First posted on arXiv 31 May 2022.

Builds on

47 of 71 references.

References · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  5. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  6. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  8. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  9. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  10. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
  11. 2020Mastering Atari with Discrete World Modelscited by 1,007
  12. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
35 more
  1. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 799
  2. 2018Universal Sentence Encodercited by 489
  3. 2018Understanding disentangling in β-VAEcited by 335
  4. 2016Deep Reinforcement Learning for Dialogue Generationcited by 288
  5. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 286
  6. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  7. 2015Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Modelscited by 207
  8. 2016A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialoguescited by 174
  9. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 173
  10. 2018The Design and Implementation of XiaoIce, an Empathetic Social Chatbotcited by 172
  11. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 166
  12. 2017Controlling Linguistic Style Aspects in Neural Language Generationcited by 149
  13. 2018Learning to Write with Cooperative Discriminatorscited by 119
  14. 2017Adversarial Learning for Neural Dialogue Generationcited by 103
  15. 2019Learning from Dialogue after Deployment: Feed Yourself, Chatbot!cited by 102
  16. 2021Representation Matters: Offline Pretraining for Sequential Decision Makingcited by 91
  17. 2020Few-shot Natural Language Generation for Task-Oriented Dialogcited by 89
  18. 2019Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Modelscited by 53
  19. 2019Do Neural Dialog Systems Use the Conversation History Effectively? An Empirical Studycited by 47
  20. 2019Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systemscited by 42
  21. 2017A Deep Reinforcement Learning Chatbotcited by 36
  22. 2022CHAI: A CHatbot AI for Task-Oriented Dialogue with Offline Reinforcement Learningcited by 34
  23. 2021How BPE Affects Memorization in Transformerscited by 29
  24. 2019Generating Empathetic Responses by Looking Ahead the User's Sentimentcited by 27
  25. 2019Hierarchical Reinforcement Learning for Open-Domain Dialogcited by 27
  26. 2017Trainable Greedy Decoding for Neural Machine Translationcited by 23
  27. 2017Hierarchical Text Generation and Planning for Strategic Dialoguecited by 23
  28. 2018Dialogue Generation: From Imitation Learning to Inverse Reinforcement Learningcited by 21
  29. 2018A Hierarchical Latent Structure for Variational Conversation Modelingcited by 17
  30. 2018Sentiment Adaptive End-to-End Dialog Systemscited by 17
  31. 2020Predictive Coding for Locally-Linear Controlcited by 17
  32. 2019Can Unconditional Language Models Recover Arbitrary Sentences?cited by 15
  33. 2016Policy Networks with Two-Stage Training for Dialogue Systemscited by 14
  34. 2018A Stable and Effective Learning Strategy for Trainable Greedy Decodingcited by 12
  35. 2016Sample-efficient Deep Reinforcement Learning for Dialog Controlcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 6; 2024: 4; 2025: 2; 2026: 2.

16 papers.

Sort:

By area:

  1. 2023Leveraging Large Language Models in Conversational Recommender Systemscited by 63
  2. 2023Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Expertscited by 29
  3. 2024Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Expertscited by 24
  4. 2023A General Theory for Softmax Gating Multinomial Logistic Mixture of Expertscited by 23
  5. 2024CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competitioncited by 23
  6. 2024On Least Square Estimation in Softmax Gating Mixture of Expertscited by 23
  7. 2023From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscapecited by 18
  8. 2024Statistical Advantages of Perturbing Cosine Router in Mixture of Expertscited by 16
  9. 2023AI Text-to-Behavior: A Study In Steerabilitycited by 11
  10. 2022DGPO: Discovering Multiple Strategies with Diversity-Guided Policy Optimizationcited by 10
  11. 2025Convergence Rates for Softmax Gating Mixture of Expertscited by 4
  12. 2023Offline Reinforcement Learning for Mixture-of-Expert Dialogue Managementcited by 3
4 more
  1. 2022Trust in Language Grounding: a new AI challenge for human-robot teamscited by 1
  2. 2025On Minimax Estimation of Parameters in Softmax-Contaminated Mixture of Expertscited by 1
  3. 2026Improving Minimax Estimation Rates for Contaminated Mixture of Multinomial Logistic Experts via Expert Heterogeneitycited by 0
  4. 2026Robust Exploration in Directed Controller Synthesis via Reinforcement Learning with Soft Mixture-of-Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Tailored Conversations beyond LLMs: A RL-Based Dialogue Manager
  2. 2025SpeakRL: Synergizing Reasoning, Speaking, and Acting in Language Models with Reinforcement Learning
  3. 2023Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents
  4. 2021Integrating Pretrained Language Model for Dialogue Policy Learning
  5. 2020Human-centric Dialog Training via Offline Reinforcement Learning
  6. 2024Interactive Dialogue Agents via Reinforcement Learning on Hindsight Regenerations
  7. 2023Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversations
  8. 2024Reinforcement Learning Problem Solving with Large Language Models