# Improving Neural Language Modeling via Adversarial Training

arXiv: https://arxiv.org/abs/1906.03805 · HTML: https://mlsanity.com/paper/1906.03805/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jun 2019.

## Built on by (most cited first, 30 of 49)

- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [A Survey on Data Augmentation for Text Classification](https://mlsanity.com/paper/2107.03158/) (2021)
- [Autoregressive Knowledge Distillation through Imitation Learning](https://mlsanity.com/paper/2009.07253/) (2020)
- [IsoScore: Measuring the Uniformity of Embedding Space Utilization](https://mlsanity.com/paper/2108.07344/) (2021)
- [Aug-NeRF: Training Stronger Neural Radiance Fields with Triple-Level Physically-Grounded Augmentations](https://mlsanity.com/paper/2207.01164/) (2022)
- [GenDet: Towards Good Generalizations for AI-Generated Image Detection](https://mlsanity.com/paper/2312.08880/) (2023)
- [BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translation](https://mlsanity.com/paper/2109.04588/) (2021)
- [Prompt Optimization via Adversarial In-Context Learning](https://mlsanity.com/paper/2312.02614/) (2023)
- [Rare Tokens Degenerate All Tokens: Improving Neural Text Generation via Adaptive Gradient Gating for Rare Token Embeddings](https://mlsanity.com/paper/2109.03127/) (2021)
- [Data-Efficient GAN Training Beyond (Just) Augmentations: A Lottery Ticket Perspective](https://mlsanity.com/paper/2103.00397/) (2021)
- [A Survey of Adversarial Defences and Robustness in NLP](https://mlsanity.com/paper/2203.06414/) (2022)
- [Rethinking Perturbations in Encoder-Decoders for Fast Training](https://mlsanity.com/paper/2104.01853/) (2021)
- [Evolving Diverse Red-team Language Models in Multi-round Multi-agent Games](https://mlsanity.com/paper/2310.00322/) (2023)
- [Adversarial Reinforced Instruction Attacker for Robust Vision-Language Navigation](https://mlsanity.com/paper/2107.11252/) (2021)
- [CAP: Co-Adversarial Perturbation on Weights and Features for Improving Generalization of Graph Neural Networks](https://mlsanity.com/paper/2110.14855/) (2021)
- [NOSMOG: Learning Noise-robust and Structure-aware MLPs on Graphs](https://mlsanity.com/paper/2208.10010/) (2022)
- [Benchmarking Robustness in Neural Radiance Fields](https://mlsanity.com/paper/2301.04075/) (2023)
- [Detecting and Understanding Vulnerabilities in Language Models via Mechanistic Interpretability](https://mlsanity.com/paper/2407.19842/) (2024)
- [Pandora's Box or Aladdin's Lamp: A Comprehensive Analysis Revealing the Role of RAG Noise in Large Language Models](https://mlsanity.com/paper/2408.13533/) (2024)
- [A Review of ChatGPT Applications in Education, Marketing, Software Engineering, and Healthcare: Benefits, Drawbacks, and Research Directions](https://mlsanity.com/paper/2305.00237/) (2023)
- [Adversarial Feature Augmentation and Normalization for Visual Recognition](https://mlsanity.com/paper/2103.12171/) (2021)
- [UniDrop: A Simple yet Effective Technique to Improve Transformer without Extra Cost](https://mlsanity.com/paper/2104.04946/) (2021)
- [Toward Adversarial Training on Contextualized Language Representation](https://mlsanity.com/paper/2305.04557/) (2023)
- [Consistency Training with Virtual Adversarial Discrete Perturbation](https://mlsanity.com/paper/2104.07284/) (2021)
- [Bi-SimCut: A Simple Strategy for Boosting Neural Machine Translation](https://mlsanity.com/paper/2206.02368/) (2022)
- [Adversarial Self-Attention for Language Understanding](https://mlsanity.com/paper/2206.12608/) (2022)
- [Adversarial Regularization as Stackelberg Game: An Unrolled Optimization Approach](https://mlsanity.com/paper/2104.04886/) (2021)
- [Towards the Worst-case Robustness of Large Language Models](https://mlsanity.com/paper/2501.19040/) (2025)
- [Retrieval-Augmented Purifier for Robust LLM-Empowered Recommendation](https://mlsanity.com/paper/2504.02458/) (2025)
- [Adversarial Training: A Survey](https://mlsanity.com/paper/2410.15042/) (2024)

## Abstract (arXiv)

Recently, substantial progress has been made in language modeling by using deep neural networks. However, in practice, large scale neural language models have been shown to be prone to overfitting. In this paper, we present a simple yet highly effective adversarial training mechanism for regularizing neural language models. The idea is to introduce adversarial noise to the output embedding layer while training the models. We show that the optimal adversarial noise yields a simple closed-form solution, thus allowing us to develop a simple and time efficient algorithm. Theoretically, we show that our adversarial mechanism effectively encourages the diversity of the embedding vectors, helping to increase the robustness of models. Empirically, we show that our method improves on the single model state-of-the-art results for language modeling on Penn Treebank (PTB) and Wikitext-2, achieving test perplexity scores of 46.01 and 38.07, respectively. When applied to machine translation, our method improves over various transformer-based translation baselines in BLEU scores on the WMT14 English-German and IWSLT14 German-English tasks.