# Single Headed Attention RNN: Stop Thinking With Your Head

arXiv: https://arxiv.org/abs/1911.11423 · HTML: https://mlsanity.com/paper/1911.11423/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Nov 2019.

## Built on by (most cited first, 26 of 26)

- [Transformers are Multi-State RNNs](https://mlsanity.com/paper/2401.06104/) (2024)
- [SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networks](https://mlsanity.com/paper/2302.13939/) (2023)
- [Cramming: Training a Language Model on a Single GPU in One Day](https://mlsanity.com/paper/2212.14034/) (2022)
- [When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute](https://mlsanity.com/paper/2102.12459/) (2021)
- [Fine-Tuning Pre-trained Transformers into Decaying Fast Weights](https://mlsanity.com/paper/2210.04243/) (2022)
- [A Generalizable Approach to Learning Optimizers](https://mlsanity.com/paper/2106.00958/) (2021)
- [A Language Model with Limited Memory Capacity Captures Interference in Human Sentence Processing](https://mlsanity.com/paper/2310.16142/) (2023)
- [A survey on attention mechanisms for medical applications: are we moving towards better algorithms?](https://mlsanity.com/paper/2204.12406/) (2022)
- [Automated essay scoring using efficient transformer-based language models](https://mlsanity.com/paper/2102.13136/) (2021)
- [Mukayese: Turkish NLP Strikes Back](https://mlsanity.com/paper/2203.01215/) (2022)
- [On "Scientific Debt" in NLP: A Case for More Rigour in Language Model Pre-Training Research](https://mlsanity.com/paper/2306.02870/) (2023)
- [SRU++: Pioneering Fast Recurrence with Attention for Speech Recognition](https://mlsanity.com/paper/2110.05571/) (2021)
- [LegaLMFiT: Efficient Short Legal Text Classification with LSTM Language Model Pre-Training](https://mlsanity.com/paper/2109.00993/) (2021)
- [Attention vs non-attention for a Shapley-based explanation method](https://mlsanity.com/paper/2104.12424/) (2021)
- [Simple Projection Variants Improve ColBERT Performance](https://mlsanity.com/paper/2510.12327/) (2025)
- [Transformer models as an efficient replacement for statistical test suites to evaluate the quality of random numbers](https://mlsanity.com/paper/2405.03904/) (2024)
- [Efficient Language Modeling for Low-Resource Settings with Hybrid RNN-Transformer Architectures](https://mlsanity.com/paper/2502.00617/) (2025)
- [Dynamic Pooling Improves Nanopore Base Calling Accuracy](https://mlsanity.com/paper/2105.07520/) (2021)
- [Online Self-Attentive Gated RNNs for Real-Time Speaker Separation](https://mlsanity.com/paper/2106.13493/) (2021)
- [SHAQ: Single Headed Attention with Quasi-Recurrence](https://mlsanity.com/paper/2108.08207/) (2021)
- [Circling Back to Recurrent Models of Language](https://mlsanity.com/paper/2211.01848/) (2022)
- [Implicit Geometry and Interaction Embeddings Improve Few-Shot Molecular Property Prediction](https://mlsanity.com/paper/2302.02055/) (2023)
- [RCMHA: Relative Convolutional Multi-Head Attention for Natural Language Modelling](https://mlsanity.com/paper/2308.03429/) (2023)
- [Attention-Based Recurrent Neural Network For Automatic Behavior Laying Hen Recognition](https://mlsanity.com/paper/2401.09880/) (2024)
- [Deep learning-based approach for tomato classification in complex scenes](https://mlsanity.com/paper/2401.15055/) (2024)
- [Building LLM Agent Systems the Deep Learning Way: From Modular Design to Architecture Search](https://mlsanity.com/paper/2610.04961/) (2026)

## Abstract (arXiv)

The leading approaches in language modeling are all obsessed with TV shows of my youth - namely Transformers and Sesame Street. Transformers this, Transformers that, and over here a bonfire worth of GPU-TPU-neuromorphic wafer scale silicon. We opt for the lazy path of old and proven techniques with a fancy crypto inspired acronym: the Single Headed Attention RNN (SHA-RNN). The author's lone goal is to show that the entire field might have evolved a different direction if we had instead been obsessed with a slightly different acronym and slightly different result. We take a previously strong language model based only on boring LSTMs and get it to within a stone's throw of a stone's throw of state-of-the-art byte level language model results on enwik8. This work has undergone no intensive hyperparameter optimization and lived entirely on a commodity desktop machine that made the author's small studio apartment far too warm in the midst of a San Franciscan summer. The final results are achievable in plus or minus 24 hours on a single GPU as the author is impatient. The attention mechanism is also readily extended to large contexts with minimal computation. Take that Sesame Street.