mlsanityData to 14 Oct 2026

SPMoE: Generate Multiple Pattern-Aware Outputs with Sparse Pattern Mixture of Experts

arXiv 2108.07535 ↗First posted 17 Aug 2021cs.CLcited by 0

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Many generation tasks follow a one-to-many mapping relationship: each input could be associated with multiple outputs. Existing methods like Conditional Variational AutoEncoder(CVAE) employ a latent variable to model this one-to-many relationship. However, this high-dimensional and dense latent variable lacks explainability and usually leads to poor and uncontrollable generations. In this paper, we innovatively introduce the linguistic concept of pattern to decompose the one-to-many mapping into multiple one-to-one mappings and further propose a model named Sparse Pattern Mixture of Experts(SPMoE). Each one-to-one mapping is associated with a conditional generation pattern and is modeled with an expert in SPMoE. To ensure each language pattern can be exclusively handled with an expert model for better explainability and diversity, a sparse mechanism is employed to coordinate all the expert models in SPMoE. We assess the performance of our SPMoE on the paraphrase generation task and the experiment results prove that SPMoE can achieve a good balance in terms of quality, pattern-level diversity, and corpus-level diversity.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 14 of its 17 references that are in the map. First posted on arXiv 17 Aug 2021.

Builds on

14 of 17 references.

Closest ancestors · 1

Same direction, newest first

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 13

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  3. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
  4. 2016SeqGAN: Sequence Generative Adversarial Nets with Policy Gradientcited by 376
  5. 2016From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classificationcited by 341
  6. 2016Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Modelscited by 301
  7. 2017Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoderscited by 222
  8. 2019Sparse Sequence-to-Sequence Modelscited by 170
  9. 2019Mixture Models for Diverse Machine Translation: Tricks of the Tradecited by 77
  10. 2017A Deep Generative Framework for Paraphrase Generationcited by 54
  11. 2018On Controllable Sparse Alternatives to Softmaxcited by 27
  12. 2018Sequence to Sequence Mixture Model for Diverse Machine Translationcited by 13
1 more
  1. 2018D-PAGE: Diverse Paraphrase Generationcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models
  2. 2025Beyond Redundancy: Diverse and Specialized Multi-Expert Sparse Autoencoder
  3. 2025Unveiling Hidden Collaboration within Mixture-of-Experts in Large Language Models
  4. 2026Decoupled Mixture-of-Experts for Parametric Knowledge Injection
  5. 2025Mosaic Pruning: A Hierarchical Framework for Generalizable Pruning of Mixture-of-Experts Models
  6. 2026DirMoE: Dirichlet-routed Mixture of Experts
  7. 2026Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
  8. 2024SimSMoE: Solving Representational Collapse via Similarity Measure