Data to 7 Oct 2026

Regularizing and Optimizing LSTM Language Models

arXiv 1708.02182 ↗First posted 7 Aug 2017cs.CL cs.LG cs.NEcited by 205

Not on the map. Why?

Abstract (arXiv)

Recurrent neural networks (RNNs), such as long short-term memory networks (LSTMs), serve as a fundamental building block for many sequence learning tasks, including machine translation, language modeling, and question answering. In this paper, we consider the specific problem of word-level language modeling and investigate strategies for regularizing and optimizing LSTM-based models. We propose the weight-dropped LSTM which uses DropConnect on hidden-to-hidden weights as a form of recurrent regularization. Further, we introduce NT-ASGD, a variant of the averaged stochastic gradient method, wherein the averaging trigger is determined using a non-monotonic condition as opposed to being tuned by the user. Using these and other regularization strategies, we achieve state-of-the-art word level perplexities on two data sets: 57.3 on Penn Treebank and 65.8 on WikiText-2. In exploring the effectiveness of a neural cache in conjunction with our proposed model, we achieve an even lower state-of-the-art perplexity of 52.8 on Penn Treebank and 52.0 on WikiText-2.

Key facts. Cited by 205 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Aug 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 68; 2022: 43; 2023: 32; 2024: 32; 2025: 17; 2026: 13.

205 papers.

Sort:

By area:

2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
2021Random Feature Attentioncited by 350
2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
2021Autoregressive Diffusion Modelscited by 208
2021R-Drop: Regularized Dropout for Neural Networkscited by 202
2021Lessons on Parameter Sharing across Layers in Transformerscited by 85
2021Gradients are Not All You Needcited by 82
2022Robustness to Unbounded Smoothness of Generalized SignSGDcited by 80
2021Language Model Evaluation Beyond Perplexitycited by 77
2024Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooMcited by 77
2022Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methodscited by 75

38 more

2021Stabilizing Equilibrium Models by Jacobian Regularizationcited by 71
2022Latent Diffusion Energy-Based Model for Interpretable Text Modelingcited by 67
2021A Comprehensive Survey on Hardware-Aware Neural Architecture Searchcited by 60
2021Randomness In Neural Network Training: Characterizing The Impact of Toolingcited by 57
2022Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averagingcited by 54
2021Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularizationcited by 48
2024From Gradient Clipping to Normalization for Heavy Tailed SGDcited by 48
2021Nondeterminism and Instability in Neural Network Optimizationcited by 47
2024Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitatecited by 46
2022History Compression via Language Models in Reinforcement Learningcited by 39
2021Evaluating Saliency Methods for Neural Language Modelscited by 37
2021AutoDropout: Learning Dropout Patterns to Regularize Deep Networkscited by 35
2021On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implicationscited by 31
2021From Machine Translation to Code-Switching: Generating High-Quality Code-Switched Textcited by 29
2023Why do Nearest Neighbor Language Models Work?cited by 29
2023Parameter-Agnostic Optimization under Relaxed Smoothnesscited by 29
2024Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Modelscited by 28
2021Rare Tokens Degenerate All Tokens: Improving Neural Text Generation via Adaptive Gradient Gating for Rare Token Embeddingscited by 27
2021GNN-LM: Language Modeling based on Global Contexts via GNNcited by 27
2022Interacting with next-phrase suggestions: How suggestion systems aid and influence the cognitive processes of writingcited by 27
2023BEND: Benchmarking DNA Language Models on biologically meaningful taskscited by 27
2021Selfish Sparse RNN Trainingcited by 26
2022Efficient recurrent architectures through activity sparsity and sparse back-propagation through timecited by 26
2021Data Augmentation in Natural Language Processing: A Novel Text Generation Approach for Long and Short Text Classifierscited by 24
2025Pre-training under infinite computecited by 24
2022Query-Efficient and Scalable Black-Box Adversarial Attacks on Discrete Sequential Data via Bayesian Optimizationcited by 22
2023Variance-reduced Clipping for Non-convex Optimizationcited by 22
2024Explaining Datasets in Words: Statistical Models with Natural Language Parameterscited by 21
2022A Communication-Efficient Distributed Gradient Clipping Algorithm for Training Deep Neural Networkscited by 20
2021CPT: Efficient Deep Neural Network Training via Cyclic Precisioncited by 19
2022GraB: Finding Provably Better Data Permutations than Random Reshufflingcited by 18
2021Rethinking Perturbations in Encoder-Decoders for Fast Trainingcited by 17
2021"BNN - BN = ?": Training Binary Neural Networks without Batch Normalizationcited by 17
2021Recurrent Neural Network from Adder's Perspective: Carry-lookahead RNNcited by 16
2021Adapting GPT, GPT-2 and BERT Language Models for Speech Recognitioncited by 16
2021Learning Hierarchical Structures with Differentiable Nondeterministic Stackscited by 16
2021GAN Computers Generate Arts? A Survey on Visual Arts, Music, and Literary Text Generation using Generative Adversarial Networkcited by 15
2021Latent Space Energy-Based Model of Symbol-Vector Coupling for Text Generation and Classificationcited by 14

50 most cited of 205 All 205 (JSON)