# Faster Transformer Decoding: N-gram Masked Self-Attention

arXiv: https://arxiv.org/abs/2001.04589 · HTML: https://mlsanity.com/paper/2001.04589/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jan 2020.

## Built on by (most cited first, 6 of 6)

- [A Survey on Transformer Compression](https://mlsanity.com/paper/2402.05964/) (2024)
- [Multilingual Neural Machine Translation with Deep Encoder and Multiple Shallow Decoders](https://mlsanity.com/paper/2206.02079/) (2022)
- [Revisiting the Markov Property for Machine Translation](https://mlsanity.com/paper/2402.02084/) (2024)
- [SimpleTRON: Simple Transformer with O(N) Complexity](https://mlsanity.com/paper/2111.15588/) (2021)
- [Chat-Ghosting: A Comparative Study of Methods for Auto-Completion in Dialog Systems](https://mlsanity.com/paper/2507.05940/) (2025)
- [Mask-Aware Execution for Efficient JEPA Training](https://mlsanity.com/paper/2609.22674/) (2026)

## Abstract (arXiv)

Motivated by the fact that most of the information relevant to the prediction of target tokens is drawn from the source sentence S=s1, ldots, sS, we propose truncating the target-side window used for computing self-attention by making an N-gram assumption. Experiments on WMT EnDe and EnFr data sets show that the N-gram masked self-attention model loses very little in BLEU score for N values in the range 4, ldots, 8, depending on the task.