mlsanityData to 7 Oct 2026

On Biasing Transformer Attention Towards Monotonicity

arXiv 2104.03945 ↗First posted 8 Apr 2021cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Many sequence-to-sequence tasks in natural language processing are roughly monotonic in the alignment between source and target sequence, and previous work has facilitated or enforced learning of monotonic attention behavior via specialized attention functions or pretraining. In this work, we introduce a monotonicity loss function that is compatible with standard attention mechanisms and test it on several sequence-to-sequence tasks: grapheme-to-phoneme conversion, morphological inflection, transliteration, and dialect normalization. Experiments show that we can achieve largely monotonic behavior. Performance is mixed, with larger gains on top of RNN baselines. General monotonicity does not benefit transformer multihead attention, however, we see isolated improvements when only a subset of heads is biased towards monotonic behavior.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 21 of its 33 references that are in the map. First posted on arXiv 8 Apr 2021.

Builds on

21 of 33 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  4. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  5. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  6. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  7. 2019Monotonic Infinite Lookback Attention for Simultaneous Machine Translationcited by 103
  8. 2019Monotonic Multihead Attentioncited by 91
  9. 2017Online and Linear-Time Attention by Enforcing Monotonic Alignmentscited by 85
  10. 2017Monotonic Chunkwise Attentioncited by 82
  11. 2019Revisiting Low-Resource Neural Machine Translation: A Case Studycited by 61
  12. 2020Applying the Transformer to Character-level Transductioncited by 43
9 more
  1. 2018Multi-Head Attention with Disagreement Regularizationcited by 35
  2. 2020Transformer based Grapheme-to-Phoneme Conversioncited by 32
  3. 2020Scheduled DropHead: A Regularization Method for Transformer Modelscited by 28
  4. 2020Optimizing Transformer for Low-Resource Neural Machine Translationcited by 24
  5. 2019Exact Hard Monotonic Attention for Character-Level Transductioncited by 22
  6. 2016Incorporating Structural Alignment Biases into an Attentional Neural Translation Modelcited by 16
  7. 2020The Sockeye 2 Neural Machine Translation Toolkit at AMTA 2020cited by 16
  8. 2018Hard Non-Monotonic Attention for Character-Level Transductioncited by 12
  9. 2020Multi-head Monotonic Chunkwise Attention For Online Speech Recognitioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 3; 2023: 0; 2024: 0; 2025: 0; 2026: 1.

5 papers.

  1. 2022Towards a Broad Coverage Named Entity Resource: A Data-Efficient Approach for Many Diverse Languagescited by 8
  2. 2021How Suitable Are Subword Segmentation Strategies for Translating Non-Concatenative Morphology?cited by 7
  3. 2022Transformers in Action: Weakly Supervised Action Segmentationcited by 4
  4. 2022Regotron: Regularizing the Tacotron2 architecture via monotonic alignment losscited by 3
  5. 2026CI-ICM: Channel Importance-driven Learned Image Coding for Machinescited by 0