# Sparse and Constrained Attention for Neural Machine Translation

arXiv: https://arxiv.org/abs/1805.08241 · HTML: https://mlsanity.com/paper/1805.08241/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 May 2018.

## Built on by (most cited first, 13 of 13)

- [Sparse Attention with Linear Units](https://mlsanity.com/paper/2104.07012/) (2021)
- [Is Sparse Attention more Interpretable?](https://mlsanity.com/paper/2106.01087/) (2021)
- [Neural Collaborative Graph Machines for Table Structure Recognition](https://mlsanity.com/paper/2111.13359/) (2021)
- [Fast, Differentiable and Sparse Top-k: a Convex Analysis Perspective](https://mlsanity.com/paper/2302.01425/) (2023)
- [Alternating Differentiation for Optimization Layers](https://mlsanity.com/paper/2210.01802/) (2022)
- [Evidential Softmax for Sparse Multimodal Distributions in Deep Generative Models](https://mlsanity.com/paper/2110.14182/) (2021)
- [Compositional Generalisation with Structured Reordering and Fertility Layers](https://mlsanity.com/paper/2210.03183/) (2022)
- [Speeding Up Entmax](https://mlsanity.com/paper/2111.06832/) (2021)
- [Adaptive Sparse and Monotonic Attention for Transformer-based Automatic Speech Recognition](https://mlsanity.com/paper/2209.15176/) (2022)
- [Memory Wrap: a Data-Efficient and Interpretable Extension to Image Classification Models](https://mlsanity.com/paper/2106.01440/) (2021)
- [Empty SPACE: Cross-Attention Sparsity for Concept Erasure in Diffusion Models](https://mlsanity.com/paper/2605.10198/) (2026)
- [Decision Machines: Congruent Decision Trees](https://mlsanity.com/paper/2101.11347/) (2021)
- [Explainability of Text Processing and Retrieval Methods: A Survey](https://mlsanity.com/paper/2212.07126/) (2022)

## Abstract (arXiv)

In NMT, words are sometimes dropped from the source or generated repeatedly in the translation. We explore novel strategies to address the coverage problem that change only the attention transformation. Our approach allocates fertilities to source words, used to bound the attention each word can receive. We experiment with various sparse and constrained attention transformations and propose a new one, constrained sparsemax, shown to be differentiable and sparse. Empirical evaluation is provided in three languages pairs.