mlsanityData to 14 Oct 2026

Improved Training of Mixture-of-Experts Language GANs

arXiv 2302.11875 ↗First posted 23 Feb 2023cs.CLcited by 7

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Despite the dramatic success in image generation, Generative Adversarial Networks (GANs) still face great challenges in synthesizing sequences of discrete elements, in particular human language. The difficulty in generator training arises from the limited representation capacity and uninformative learning signals obtained from the discriminator. In this work, we (1) first empirically show that the mixture-of-experts approach is able to enhance the representation capacity of the generator for language GANs and (2) harness the Feature Statistics Alignment (FSA) paradigm to render fine-grained learning signals to advance the generator training. Specifically, FSA forces the mean statistics of the distribution of fake data to approach that of real samples as close as possible in the finite-dimensional feature space. Empirical study on synthetic and real benchmarks shows the superior performance in quantitative evaluation and demonstrates the effectiveness of our approach to adversarial text generation.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 26 of its 31 references that are in the map. First posted on arXiv 23 Feb 2023.

Builds on

26 of 31 references.

References · 26

  1. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  2. 2017Improved Training of Wasserstein GANscited by 2,235
  3. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  4. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,127
  5. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
  6. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 491
  7. 2016SeqGAN: Sequence Generative Adversarial Nets with Policy Gradientcited by 376
  8. 2018The relativistic discriminator: a key element missing from standard GANcited by 238
  9. 2014A* Samplingcited by 151
  10. 2017Adversarial Learning for Neural Dialogue Generationcited by 103
  11. 2017Long Text Generation via Adversarial Training with Leaked Informationcited by 100
  12. 2018MaskGAN: Better Text Generation via Filling in the______cited by 89
14 more
  1. 2018Relational recurrent neural networkscited by 65
  2. 2017Multi-Agent Diverse Generative Adversarial Networkscited by 59
  3. 2017Adversarial Feature Matching for Text Generationcited by 58
  4. 2017Adversarial Ranking for Language Generationcited by 50
  5. 2016GANS for Sequences of Discrete Elements with the Gumbel-softmax Distributioncited by 48
  6. 2017Maximum-Likelihood Augmented Discrete Generative Adversarial Networkscited by 39
  7. 2019Training language GANs from Scratchcited by 31
  8. 2018Adversarial Text Generation via Feature-Mover's Distancecited by 27
  9. 2019CatGAN: Category-aware Generative Adversarial Networks with Hierarchical Evolutionary Learning for Category Text Generationcited by 16
  10. 2020ColdGANs: Taming Language GANs with Cautious Sampling Strategiescited by 12
  11. 2020Self-Adversarial Learning with Comparative Discrimination for Text Generationcited by 11
  12. 2019ARAML: A Stable Adversarial Training Framework for Text Generationcited by 9
  13. 2019Multi-agent Learning for Neural Machine Translationcited by 5
  14. 2018Distilling Knowledge for Search-based Structured Predictioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 3; 2026: 1.

7 papers.

  1. 2023Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizercited by 29
  2. 2025Mixture of Experts (MoE): A Big Data Perspectivecited by 18
  3. 2025CodeMixBench: Evaluating Code-Mixing Capabilities of LLMs Across 18 Languagescited by 10
  4. 2024Integration of Mixture of Experts and Multimodal Generative AI in Internet of Vehicles: A Surveycited by 6
  5. 2024Utilizing Local Hierarchy with Adversarial Training for Hierarchical Text Classificationcited by 3
  6. 2025FLAME: Towards Federated Fine-Tuning Large Language Models Through Adaptive SMoEcited by 0
  7. 2026Enhanced Mixture 3D CGAN for Completion and Generation of 3D Objectscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2018MEGAN: Mixture of Experts of Generative Adversarial Networks for Multimodal Image Generation
  2. 2023Exploring Sparse MoE in GANs for Text-conditioned Image Synthesis
  3. 2019Hierarchical Mixtures of Generators for Adversarial Learning
  4. 2025Generative Adversarial Networks Bridging Art and Machine Intelligence
  5. 2020TextGAIL: Generative Adversarial Imitation Learning for Text Generation
  6. 2023Ten Years of Generative Adversarial Nets (GANs): A survey of the state-of-the-art
  7. 2023StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis
  8. 2024A Survey on Mixture of Experts in Large Language Models