mlsanityData to 7 Oct 2026

Divide, Conquer, and Combine: Mixture of Semantic-Independent Experts for Zero-Shot Dialogue State Tracking

arXiv 2306.00434 ↗First posted 1 Jun 2023cs.CLcited by 10

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Zero-shot transfer learning for Dialogue State Tracking (DST) helps to handle a variety of task-oriented dialogue domains without the cost of collecting in-domain data. Existing works mainly study common data- or model-level augmentation methods to enhance the generalization but fail to effectively decouple the semantics of samples, limiting the zero-shot performance of DST. In this paper, we present a simple and effective "divide, conquer and combine" solution, which explicitly disentangles the semantics of seen data, and leverages the performance and robustness with the mixture-of-experts mechanism. Specifically, we divide the seen data into semantically independent subsets and train corresponding experts, the newly unseen samples are mapped and inferred with mixture-of-experts with our designed ensemble inference. Extensive experiments on MultiWOZ2.1 upon the T5-Adapter show our schema significantly and consistently improves the zero-shot performance, achieving the SOTA on settings without external knowledge, with only 10M trainable parameters1.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 40 of its 54 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

40 of 54 references.

References · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  6. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 760
  7. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  8. 2019Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Datasetcited by 426
  9. 2019Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systemscited by 215
  10. 2023Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERTcited by 155
  11. 2020Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?cited by 147
  12. 2023Towards Making the Most of ChatGPT for Machine Translationcited by 138
28 more
  1. 2020Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architecturescited by 95
  2. 2021Dialogue State Tracking with a Language Model using Schema-Driven Promptingcited by 94
  3. 2022SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapterscited by 83
  4. 2020Zero-Shot Transfer Learning with Synthesized Data for Multi-Domain Dialogue State Trackingcited by 65
  5. 2019SUMBT: Slot-Utterance Matching for Universal and Scalable Belief Trackingcited by 63
  6. 2021Leveraging Slot Descriptions for Zero-Shot Cross-Domain Dialogue State Trackingcited by 61
  7. 2021Semantics Disentangling for Generalized Zero-Shot Learningcited by 56
  8. 2021Zero-Shot Dialogue State Tracking via Cross-Task Transfercited by 52
  9. 2022In-Context Learning for Few-Shot Dialogue State Trackingcited by 47
  10. 2018Global-Locally Self-Attentive Dialogue State Trackercited by 45
  11. 2020CoCo: Controllable Counterfactuals for Evaluating Dialogue State Trackerscited by 43
  12. 2020SlotRefine: A Fast Non-Autoregressive Model for Joint Intent Detection and Slot Fillingcited by 42
  13. 2020From Machine Reading Comprehension to Dialogue State Tracking: Bridging the Gapcited by 40
  14. 2019BERT-DST: Scalable End-to-End Dialogue State Tracking with Bidirectional Encoder Representations from Transformercited by 37
  15. 2021Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translationcited by 34
  16. 2022PANDA: Prompt Transfer Meets Knowledge Distillation for Efficient Model Adaptationcited by 31
  17. 2021Slot Self-Attentive Dialogue State Trackingcited by 27
  18. 2020MA-DST: Multi-Attention Based Scalable Dialog State Trackingcited by 24
  19. 2022Toward Efficient Language Model Pretraining and Downstream Adaptation via Self-Evolution: A Case Study on SuperGLUEcited by 24
  20. 2022Dialogue Summaries as Dialogue States (DS2), Template-Guided Summarization for Few-shot Dialogue State Trackingcited by 21
  21. 2017Zero-Shot Recognition using Dual Visual-Semantic Mapping Pathscited by 20
  22. 2022Dynamic Schema Graph Fusion Network for Multi-Domain Dialogue State Trackingcited by 18
  23. 2021Single-dataset Experts for Multi-dataset Question Answeringcited by 15
  24. 2021Disentangling Semantic-to-visual Confusion for Zero-shot Learningcited by 11
  25. 2021The USYD-JD Speech Translation System for IWSLT 2021cited by 11
  26. 2023Bag of Tricks for Effective Language Model Pretraining and Downstream Adaptation: A Case Study on GLUEcited by 11
  27. 2019The University of Sydney's Machine Translation System for WMT19cited by 9
  28. 2021Bridging the Gap Between Clean Data Training and Real-World Inference for Spoken Language Understandingcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 4; 2025: 3; 2026: 1.

10 papers.

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  2. 2023Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizercited by 28
  3. 2024Enhancing Dialogue State Tracking Models through LLM-backed User-Agents Simulationcited by 24
  4. 2023OrchestraLLM: Efficient Orchestration of Language Models for Dialogue State Trackingcited by 17
  5. 2025PRINCIPLES: Synthetic Strategy Memory for Proactive Dialogue Agentscited by 15
  6. 2024Diverse and Effective Synthetic Data Generation for Adaptable Zero-Shot Dialogue State Trackingcited by 6
  7. 2024Distilled Transformers with Locally Enhanced Global Representations for Face Forgery Detectioncited by 4
  8. 2024MoPE: Mixture of Prefix Experts for Zero-Shot Dialogue State Trackingcited by 1
  9. 2025HiCoLoRA: Addressing Context-Prompt Misalignment via Hierarchical Collaborative LoRA for Zero-Shot DSTcited by 0
  10. 2026Mastering the Minority: An Uncertainty-guided Multi-Expert Framework for Challenging-tailed Sequence Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Inference is All You Need: Self Example Retriever for Cross-domain Dialogue State Tracking with ChatGPT
  2. 2024SynthDST: Synthetic Data is All You Need for Few-Shot Dialog State Tracking
  3. 2024Large Language Models as Zero-shot Dialogue State Tracker through Function Calling
  4. 2024A Two-dimensional Zero-shot Dialogue State Tracking Evaluation Method using GPT-4
  5. 2023Diverse Retrieval-Augmented In-Context Learning for Dialogue State Tracking
  6. 2026GEM: Graph-Enhanced Mixture-of-Experts with ReAct Agents for Dialogue State Tracking
  7. 2024Benchmarks Underestimate the Readiness of Multi-lingual Dialogue Agents
  8. 2024Many Hands Make Light Work: Task-Oriented Dialogue System with Module-Based Mixture-of-Experts