# Improved Training of Mixture-of-Experts Language GANs

arXiv: https://arxiv.org/abs/2302.11875 · HTML: https://mlsanity.com/paper/2302.11875/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in MoE large language models, one of 12 directions in Mixture-of-experts.
- Builds on 26 of its 31 references that are in the map.
- First posted on arXiv 23 Feb 2023.

## Placed in

- Mixture-of-experts / [MoE large language models](https://mlsanity.com/moe/moe-large-language-models/)

## Builds on (most cited first, 26 of 26)

- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Texygen: A Benchmarking Platform for Text Generation Models](https://mlsanity.com/paper/1802.01886/) (2018)
- [SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient](https://mlsanity.com/paper/1609.05473/) (2016)
- [The relativistic discriminator: a key element missing from standard GAN](https://mlsanity.com/paper/1807.00734/) (2018)
- [A* Sampling](https://mlsanity.com/paper/1411.0030/) (2014)
- [Adversarial Learning for Neural Dialogue Generation](https://mlsanity.com/paper/1701.06547/) (2017)
- [Long Text Generation via Adversarial Training with Leaked Information](https://mlsanity.com/paper/1709.08624/) (2017)
- [MaskGAN: Better Text Generation via Filling in the______](https://mlsanity.com/paper/1801.07736/) (2018)
- [Relational recurrent neural networks](https://mlsanity.com/paper/1806.01822/) (2018)
- [Multi-Agent Diverse Generative Adversarial Networks](https://mlsanity.com/paper/1704.02906/) (2017)
- [Adversarial Feature Matching for Text Generation](https://mlsanity.com/paper/1706.03850/) (2017)
- [Adversarial Ranking for Language Generation](https://mlsanity.com/paper/1705.11001/) (2017)
- [GANS for Sequences of Discrete Elements with the Gumbel-softmax Distribution](https://mlsanity.com/paper/1611.04051/) (2016)
- [Maximum-Likelihood Augmented Discrete Generative Adversarial Networks](https://mlsanity.com/paper/1702.07983/) (2017)
- [Training language GANs from Scratch](https://mlsanity.com/paper/1905.09922/) (2019)
- [Adversarial Text Generation via Feature-Mover's Distance](https://mlsanity.com/paper/1809.06297/) (2018)
- [CatGAN: Category-aware Generative Adversarial Networks with Hierarchical Evolutionary Learning for Category Text Generation](https://mlsanity.com/paper/1911.06641/) (2019)
- [ColdGANs: Taming Language GANs with Cautious Sampling Strategies](https://mlsanity.com/paper/2006.04643/) (2020)
- [Self-Adversarial Learning with Comparative Discrimination for Text Generation](https://mlsanity.com/paper/2001.11691/) (2020)
- [ARAML: A Stable Adversarial Training Framework for Text Generation](https://mlsanity.com/paper/1908.07195/) (2019)
- [Multi-agent Learning for Neural Machine Translation](https://mlsanity.com/paper/1909.01101/) (2019)
- [Distilling Knowledge for Search-based Structured Prediction](https://mlsanity.com/paper/1805.11224/) (2018)

## Built on by (most cited first, 7 of 7)

- [Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer](https://mlsanity.com/paper/2310.09762/) (2023)
- [Mixture of Experts (MoE): A Big Data Perspective](https://mlsanity.com/paper/2501.16352/) (2025)
- [CodeMixBench: Evaluating Code-Mixing Capabilities of LLMs Across 18 Languages](https://mlsanity.com/paper/2507.18791/) (2025)
- [Integration of Mixture of Experts and Multimodal Generative AI in Internet of Vehicles: A Survey](https://mlsanity.com/paper/2404.16356/) (2024)
- [Utilizing Local Hierarchy with Adversarial Training for Hierarchical Text Classification](https://mlsanity.com/paper/2402.18825/) (2024)
- [FLAME: Towards Federated Fine-Tuning Large Language Models Through Adaptive SMoE](https://mlsanity.com/paper/2506.16600/) (2025)
- [Enhanced Mixture 3D CGAN for Completion and Generation of 3D Objects](https://mlsanity.com/paper/2602.08046/) (2026)

## Abstract (arXiv)

Despite the dramatic success in image generation, Generative Adversarial Networks (GANs) still face great challenges in synthesizing sequences of discrete elements, in particular human language. The difficulty in generator training arises from the limited representation capacity and uninformative learning signals obtained from the discriminator. In this work, we (1) first empirically show that the mixture-of-experts approach is able to enhance the representation capacity of the generator for language GANs and (2) harness the Feature Statistics Alignment (FSA) paradigm to render fine-grained learning signals to advance the generator training. Specifically, FSA forces the mean statistics of the distribution of fake data to approach that of real samples as close as possible in the finite-dimensional feature space. Empirical study on synthetic and real benchmarks shows the superior performance in quantitative evaluation and demonstrates the effectiveness of our approach to adversarial text generation.