# Language Models with Transformers

arXiv: https://arxiv.org/abs/1904.09408 · HTML: https://mlsanity.com/paper/1904.09408/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Apr 2019.

## Built on by (most cited first, 30 of 36)

- [Deep Neural Networks and Tabular Data: A Survey](https://mlsanity.com/paper/2110.01889/) (2021)
- [Towards Understanding and Mitigating Social Biases in Language Models](https://mlsanity.com/paper/2106.13219/) (2021)
- [Open World Compositional Zero-Shot Learning](https://mlsanity.com/paper/2101.12609/) (2021)
- [Learning Graph Embeddings for Open World Compositional Zero-Shot Learning](https://mlsanity.com/paper/2105.01017/) (2021)
- [Neural Retrievers are Biased Towards LLM-Generated Content](https://mlsanity.com/paper/2310.20501/) (2023)
- [Algorithmic progress in language models](https://mlsanity.com/paper/2403.05812/) (2024)
- [History, Development, and Principles of Large Language Models-An Introductory Survey](https://mlsanity.com/paper/2402.06853/) (2024)
- [Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directions](https://mlsanity.com/paper/2501.04437/) (2025)
- [Decoding ChatGPT: A Taxonomy of Existing Research, Current Challenges, and Possible Future Directions](https://mlsanity.com/paper/2307.14107/) (2023)
- [Improving the Efficiency of Transformers for Resource-Constrained Devices](https://mlsanity.com/paper/2106.16006/) (2021)
- [Revisiting Simple Neural Probabilistic Language Models](https://mlsanity.com/paper/2104.03474/) (2021)
- [Put Teacher in Student's Shoes: Cross-Distillation for Ultra-compact Model Compression Framework](https://mlsanity.com/paper/2507.04636/) (2025)
- [Look Ahead or Look Around? A Theoretical Comparison Between Autoregressive and Masked Pretraining](https://mlsanity.com/paper/2407.00935/) (2024)
- [Video Violence Recognition and Localization Using a Semi-Supervised Hard Attention Model](https://mlsanity.com/paper/2202.02212/) (2022)
- [Disentangling Structure and Style: Political Bias Detection in News by Inducing Document Hierarchy](https://mlsanity.com/paper/2304.02247/) (2023)
- [Fisher Information-based Efficient Curriculum Federated Learning with Large Language Models](https://mlsanity.com/paper/2410.00131/) (2024)
- [Temporal Attention Augmented Transformer Hawkes Process](https://mlsanity.com/paper/2112.14472/) (2021)
- [Hydra: A System for Large Multi-Model Deep Learning](https://mlsanity.com/paper/2110.08633/) (2021)
- [Universal Transformer Hawkes Process with Adaptive Recursive Iteration](https://mlsanity.com/paper/2112.14479/) (2021)
- [TopNet: Learning from Neural Topic Model to Generate Long Stories](https://mlsanity.com/paper/2112.07259/) (2021)
- [Self-Admitted Technical Debt Detection Approaches: A Decade Systematic Review](https://mlsanity.com/paper/2312.15020/) (2023)
- [Exploring Gen-AI applications in building research and industry: A review](https://mlsanity.com/paper/2410.01098/) (2024)
- [Variational Autoencoder with Disentanglement Priors for Low-Resource Task-Specific Natural Language Generation](https://mlsanity.com/paper/2202.13363/) (2022)
- [Finding the Pillars of Strength for Multi-Head Attention](https://mlsanity.com/paper/2305.14380/) (2023)
- [Controllable Path of Destruction](https://mlsanity.com/paper/2305.18553/) (2023)
- [More for Less: Compact Convolutional Transformers Enable Robust Medical Image Classification with Limited Data](https://mlsanity.com/paper/2307.00213/) (2023)
- [Delving Deeper Into Astromorphic Transformers](https://mlsanity.com/paper/2312.10925/) (2023)
- [Acoustic Model Optimization over Multiple Data Sources: Merging and Valuation](https://mlsanity.com/paper/2410.15620/) (2024)
- [Relational Weight Priors in Neural Networks for Abstract Pattern Learning and Language Modelling](https://mlsanity.com/paper/2103.06198/) (2021)
- [Dispatcher: A Message-Passing Approach To Language Modelling](https://mlsanity.com/paper/2105.03994/) (2021)

## Abstract (arXiv)

The Transformer architecture is superior to RNN-based models in computational efficiency. Recently, GPT and BERT demonstrate the efficacy of Transformer models on various NLP tasks using pre-trained language models on large-scale corpora. Surprisingly, these Transformer architectures are suboptimal for language model itself. Neither self-attention nor the positional encoding in the Transformer is able to efficiently incorporate the word-level sequential context crucial to language modeling. In this paper, we explore effective Transformer architectures for language model, including adding additional LSTM layers to better capture the sequential context while still keeping the computation efficient. We propose Coordinate Architecture Search (CAS) to find an effective architecture through iterative refinement of the model. Experimental results on the PTB, WikiText-2, and WikiText-103 show that CAS achieves perplexities between 20.42 and 34.11 on all problems, i.e. on average an improvement of 12.0 perplexity units compared to state-of-the-art LSTMs. The source code is publicly available.