mlsanityData to 14 Oct 2026

Can the Transformer Be Used as a Drop-in Replacement for RNNs in Text-Generating GANs?

arXiv 2108.12275 ↗First posted 26 Aug 2021cs.LG cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

In this paper we address the problem of fine-tuned text generation with a limited computational budget. For that, we use a well-performing text generative adversarial network (GAN) architecture - Diversity-Promoting GAN (DPGAN), and attempted a drop-in replacement of the LSTM layer with a self-attention-based Transformer layer in order to leverage their efficiency. The resulting Self-Attention DPGAN (SADPGAN) was evaluated for performance, quality and diversity of generated text and stability. Computational experiments suggested that a transformer architecture is unable to drop-in replace the LSTM layer, under-performing during the pre-training phase and undergoing a complete mode collapse during the GAN tuning phase. Our results suggest that the transformer architecture need to be adapted before it can be used as a replacement for RNNs in text-generating GANs.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 34 of its 50 references that are in the map. First posted on arXiv 26 Aug 2021.

Builds on

34 of 50 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  6. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  8. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  11. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  12. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
22 more
  1. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  2. 2018Self-Attention Generative Adversarial Networkscited by 809
  3. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
  4. 2015Generating Sentences from a Continuous Spacecited by 612
  5. 2015A Neural Algorithm of Artistic Stylecited by 483
  6. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 469
  7. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  8. 2020Understanding the Difficulty of Training Transformerscited by 210
  9. 2020Social Biases in NLP Models as Barriers for Persons with Disabilitiescited by 182
  10. 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
  11. 2021Generative Adversarial Transformerscited by 120
  12. 2018Training Tips for the Transformer Modelcited by 85
  13. 2015How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?cited by 78
  14. 2020Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformerscited by 67
  15. 2018DP-GAN: Diversity-Promoting Generative Adversarial Network for Generating Informative and Diversified Textcited by 40
  16. 2017Image Generation from Sketch Constraint Using Contextual GANcited by 34
  17. 2019Training language GANs from Scratchcited by 31
  18. 2020A Simple Fine-tuning Is All You Need: Towards Robust Deep Learning Via Adversarial Fine-tuningcited by 30
  19. 2021Noise Stability Regularization for Improving BERT Fine-tuningcited by 29
  20. 2019CatGAN: Category-aware Generative Adversarial Networks with Hierarchical Evolutionary Learning for Category Text Generationcited by 16
  21. 2020Pay Attention when Requiredcited by 8
  22. 2018SALSA-TEXT : self attentive latent space based adversarial text generationcited by 3

Built on by

Cited by year

Citing papers per year: 2023: 2.

2 papers.

  1. 2023Fundamentals of Generative Large Language Models and Perspectives in Cyber-Defensecited by 13
  2. 2023Stochastic Parrots Looking for Stochastic Parrots: LLMs are Easy to Fine-Tune and Hard to Detect with other LLMscited by 8