mlsanityData to 7 Oct 2026

Generative Adversarial Training Can Improve Neural Language Models

arXiv 2211.09728 ↗First posted 2 Nov 2022cs.CL cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

While deep learning in the form of recurrent neural networks (RNNs) has caused a significant improvement in neural language modeling, the fact that they are extremely prone to overfitting is still a mainly unresolved issue. In this paper we propose a regularization method based on generative adversarial networks (GANs) and adversarial training (AT), that can prevent overfitting in neural language models. Unlike common adversarial training methods such as the fast gradient sign method (FGSM) that require a second back-propagation through time, and therefore effectively require at least twice the amount of time for regular training, the overhead of our method does not exceed more than 20% of the training of the baselines.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 18 of its 20 references that are in the map. First posted on arXiv 2 Nov 2022.

Builds on

18 of 20 references.

References · 18

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  4. 2016Pointer Sentinel Mixture Modelscited by 2,992
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  6. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  7. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  8. 2016Adversarial Machine Learning at Scalecited by 934
  9. 2019Adversarial Training for Free!cited by 638
  10. 2019Adversarial Examples Improve Image Recognitioncited by 283
  11. 2016Adversarial Training Methods for Semi-Supervised Text Classificationcited by 280
  12. 2017Regularizing and Optimizing LSTM Language Modelscited by 205
6 more
  1. 2019You Only Propagate Once: Accelerating Adversarial Training via Maximal Principlecited by 181
  2. 2015A Theoretically Grounded Application of Dropout in Recurrent Neural Networkscited by 161
  3. 2016Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modelingcited by 87
  4. 2018An Analysis of Neural Language Modeling at Multiple Scalescited by 31
  5. 2017Robust Multilingual Part-of-Speech Tagging via Adversarial Trainingcited by 21
  6. 2017Fraternal Dropoutcited by 14

Built on by

Cited by year

Citing papers per year: 2023: 2.

2 papers.

  1. 2023A Review of ChatGPT Applications in Education, Marketing, Software Engineering, and Healthcare: Benefits, Drawbacks, and Research Directionscited by 12
  2. 2023Adversarial Fine-Tuning of Language Models: An Iterative Optimisation Approach for the Generation and Detection of Problematic Contentcited by 7