# Transformer on a Diet

arXiv: https://arxiv.org/abs/2002.06170 · HTML: https://mlsanity.com/paper/2002.06170/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Feb 2020.

## Built on by (most cited first, 4 of 4)

- [A Practical Survey on Faster and Lighter Transformers](https://mlsanity.com/paper/2103.14636/) (2021)
- [Adaptive Multi-Resolution Attention with Linear Complexity](https://mlsanity.com/paper/2108.04962/) (2021)
- [NiNformer: A Network in Network Transformer with Token Mixing Generated Gating Function](https://mlsanity.com/paper/2403.02411/) (2024)
- [Activator: GLU Activation Function as the Core Component of a Vision Transformer](https://mlsanity.com/paper/2405.15953/) (2024)

## Abstract (arXiv)

Transformer has been widely used thanks to its ability to capture sequence information in an efficient way. However, recent developments, such as BERT and GPT-2, deliver only heavy architectures with a focus on effectiveness. In this paper, we explore three carefully-designed light Transformer architectures to figure out whether the Transformer with less computations could produce competitive results. Experimental results on language model benchmark datasets hint that such trade-off is promising, and the light Transformer reduces 70% parameters at best, while obtains competitive perplexity compared to standard Transformer. The source code is publicly available.