# An Analysis of Neural Language Modeling at Multiple Scales

arXiv: https://arxiv.org/abs/1803.08240 · HTML: https://mlsanity.com/paper/1803.08240/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Mar 2018.

## Built on by (most cited first, 30 of 31)

- [Machine Unlearning of Features and Labels](https://mlsanity.com/paper/2108.11577/) (2021)
- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [Your Transformer May Not be as Powerful as You Expect](https://mlsanity.com/paper/2205.13401/) (2022)
- [Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformers](https://mlsanity.com/paper/2101.00234/) (2021)
- [A Cognitive Regularizer for Language Modeling](https://mlsanity.com/paper/2105.07144/) (2021)
- [Linear Log-Normal Attention with Unbiased Concentration](https://mlsanity.com/paper/2311.13541/) (2023)
- [Low-Resource Language Modelling of South African Languages](https://mlsanity.com/paper/2104.00772/) (2021)
- [Revisiting Simple Neural Probabilistic Language Models](https://mlsanity.com/paper/2104.03474/) (2021)
- [Exploring Self-Identified Counseling Expertise in Online Support Forums](https://mlsanity.com/paper/2106.12976/) (2021)
- [Working Memory Connections for LSTM](https://mlsanity.com/paper/2109.00020/) (2021)
- [Language Modelling via Learning to Rank](https://mlsanity.com/paper/2110.06961/) (2021)
- [Membership Inference Attacks Against Fine-tuned Diffusion Language Models](https://mlsanity.com/paper/2601.20125/) (2026)
- [Training and Generating Neural Networks in Compressed Weight Space](https://mlsanity.com/paper/2112.15545/) (2021)
- [Towards Zero-shot Language Modeling](https://mlsanity.com/paper/2108.03334/) (2021)
- [N-gram Is Back: Residual Learning of Neural Text Generation with n-gram Language Model](https://mlsanity.com/paper/2210.14431/) (2022)
- [Accelerating LLM Inference with Flexible N:M Sparsity via A Fully Digital Compute-in-Memory Accelerator](https://mlsanity.com/paper/2504.14365/) (2025)
- [GPUTOK: GPU Accelerated Byte Level BPE Tokenization](https://mlsanity.com/paper/2603.02597/) (2026)
- [Predicting the utility of search spaces for black-box optimization: a simple, budget-aware approach](https://mlsanity.com/paper/2112.08250/) (2021)
- [Generative Adversarial Training Can Improve Neural Language Models](https://mlsanity.com/paper/2211.09728/) (2022)
- [Consistency Deep Equilibrium Models](https://mlsanity.com/paper/2602.03024/) (2026)
- [Adaptive Block Diffusion: Resolving Training-Inference Mismatch in Diffusion Language Models](https://mlsanity.com/paper/2606.29275/) (2026)
- [Contextual Text Embeddings for Twi](https://mlsanity.com/paper/2103.15963/) (2021)
- [Minimal Effective Theory for Phonotactic Memory: Capturing Local Correlations due to Errors in Speech](https://mlsanity.com/paper/2309.02466/) (2023)
- [Efficient Language Modeling for Low-Resource Settings with Hybrid RNN-Transformer Architectures](https://mlsanity.com/paper/2502.00617/) (2025)
- [Support Tokens, Stability Margins, and a New Foundation for Robust LLMs](https://mlsanity.com/paper/2602.22271/) (2026)
- [Impact of Gender Debiased Word Embeddings in Language Modeling](https://mlsanity.com/paper/2105.00908/) (2021)
- [Optimize_Prime@DravidianLangTech-ACL2022: Emotion Analysis in Tamil](https://mlsanity.com/paper/2204.09087/) (2022)
- [Optimize_Prime@DravidianLangTech-ACL2022: Abusive Comment Detection in Tamil](https://mlsanity.com/paper/2204.09675/) (2022)
- [Estimating the Effects of Sample Training Orders for Large Language Models without Retraining](https://mlsanity.com/paper/2505.22042/) (2025)
- [EG-MLA: Embedding-Gated Multi-head Latent Attention for Scalable and Efficient LLMs](https://mlsanity.com/paper/2509.16686/) (2025)

## Abstract (arXiv)

Many of the leading approaches in language modeling introduce novel, complex and specialized architectures. We take existing state-of-the-art word level language models based on LSTMs and QRNNs and extend them to both larger vocabularies as well as character-level granularity. When properly tuned, LSTMs and QRNNs achieve state-of-the-art results on character-level (Penn Treebank, enwik8) and word-level (WikiText-103) datasets, respectively. Results are obtained in only 12 hours (WikiText-103) to 2 days (enwik8) using a single modern GPU.