mlsanityData to 7 Oct 2026

Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation

arXiv 2002.10260 ↗First posted 24 Feb 2020cs.CLcited by 42

Not on the map. Why?

Abstract (arXiv)

Transformer-based models have brought a radical change to neural machine translation. A key feature of the Transformer architecture is the so-called multi-head attention mechanism, which allows the model to focus simultaneously on different parts of the input. However, recent works have shown that most attention heads learn simple, and often redundant, positional patterns. In this paper, we propose to replace all but one attention head of each encoder layer with simple fixed -- non-learnable -- attentive patterns that are solely based on position and do not require any external knowledge. Our experiments with different data sizes and multiple language pairs show that fixing the attention heads on the encoder side of the Transformer at training time does not impact the translation quality and even increases BLEU scores by up to 3 points in low-resource scenarios.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 16; 2022: 12; 2023: 8; 2024: 3; 2025: 2; 2026: 1.

42 papers.

Sort:

By area:

  1. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  2. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  3. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  4. 2022ETSformer: Exponential Smoothing Transformers for Time-series Forecastingcited by 146
  5. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  6. 2022Unveiling Transformers with LEGO: a synthetic reasoning taskcited by 105
  7. 2021Learning Graph Structures with Transformer for Multivariate Time Series Anomaly Detection in IoTcited by 104
  8. 2021Sparse Attention with Linear Unitscited by 41
  9. 2021Leveraging redundancy in attention with Reuse Transformerscited by 33
  10. 2022Visualizing and Understanding Patch Interactions in Vision Transformercited by 26
  11. 2023Faster Causal Attention Over Large Sequences Through Sparse Flash Attentioncited by 24
  12. 2023Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Casescited by 23
30 more
  1. 2022Exploiting Multiple Sequence Lengths in Fast End to End Training for Image Captioningcited by 18
  2. 2022LSG Attention: Extrapolation of pretrained Transformers to long sequencescited by 16
  3. 2021The heads hypothesis: A unifying statistical approach towards understanding multi-headed attention in BERTcited by 15
  4. 2024Enhancing EEG-to-Text Decoding through Transferable Representations from Pre-trained Contrastive EEG-Text Masked Autoencodercited by 15
  5. 2021Predicting Attention Sparsity in Transformerscited by 13
  6. 2022Sparse Mixers: Combining MoE and Mixing to build a more efficient BERTcited by 12
  7. 2021Convolutions and Self-Attention: Re-interpreting Relative Positions in Pre-trained Language Modelscited by 11
  8. 2021Attention Weights in Transformer NMT Fail Aligning Words Between Sequences but Largely Explain Model Predictionscited by 11
  9. 2021When in Doubt, Summon the Titans: Efficient Inference with Large Modelscited by 8
  10. 2021Snowflake: Scaling GNNs to High-Dimensional Continuous Control via Parameter Freezingcited by 7
  11. 2021Eigen Analysis of Self-Attention and its Reconstruction from Partial Computationcited by 6
  12. 2021Human Guided Exploitation of Interpretable Attention Patterns in Summarization and Topic Segmentationcited by 6
  13. 2022Paying More Attention to Self-attention: Improving Pre-trained Language Models via Attention Guidingcited by 6
  14. 2025Beyond Statistical Learning: Exact Learning Is Essential for General Intelligencecited by 6
  15. 2025Dimensional Collapse in Transformer Attention Outputs: A Challenge for Sparse Dictionary Learningcited by 6
  16. 2022Fast-FNet: Accelerating Transformer Encoder Models via Efficient Fourier Layerscited by 5
  17. 2024Only Send What You Need: Learning to Communicate Efficiently in Federated Multilingual Machine Translationcited by 2
  18. 2022Tri-Attention: Explicit Context-Aware Attention Mechanism for Natural Language Processingcited by 1
  19. 2023Advancing Text-to-GLOSS Neural Translation Using a Novel Hyper-parameter Optimization Techniquecited by 1
  20. 2021To Understand Representation of Layer-aware Sequence Encoders as Multi-order-graphcited by 0
  21. 2021Explainable Enterprise Credit Rating via Deep Feature Crossing Networkcited by 0
  22. 2021Exploring Low-Cost Transformer Model Compression for Large-Scale Commercial Reply Suggestionscited by 0
  23. 2022Attention Mechanism with Energy-Friendly Operationscited by 0
  24. 2023Attention Link: An Efficient Attention-Based Low Resource Machine Translation Architecturecited by 0
  25. 2023On Inductive Biases for Machine Learning in Data Constrained Settingscited by 0
  26. 2023Machine Translation by Projecting Text into the Same Phonetic-Orthographic Space Using a Common Encodingcited by 0
  27. 2023Accelerating Vision Transformers Based on Heterogeneous Attention Patternscited by 0
  28. 2023Picking the Underused Heads: A Network Pruning Perspective of Attention Head Selection for Fusing Dialogue Coreference Informationcited by 0
  29. 2024Localized Gaussians as Self-Attention Weights for Point Clouds Correspondencecited by 0
  30. 2026When Can Attention Heads Be Statically Defined?cited by 0