mlsanityData to 7 Oct 2026

The Case for Translation-Invariant Self-Attention in Transformer-Based Language Models

arXiv 2106.01950 ↗First posted 3 Jun 2021cs.CL cs.AI cs.LGcited by 23

Not on the map. Why?

Abstract (arXiv)

Mechanisms for encoding positional information are central for transformer-based language models. In this paper, we analyze the position embeddings of existing language models, finding strong evidence of translation invariance, both for the embeddings themselves and for their effect on self-attention. The degree of translation invariance increases during training and correlates positively with model performance. Our findings lead us to propose translation-invariant self-attention (TISA), which accounts for the relative position between tokens in an interpretable fashion without needing conventional position embeddings. Our proposal has several theoretical advantages over existing position-representation approaches. Experiments show that it improves on regular ALBERT on GLUE tasks, while only adding orders of magnitude less positional parameters.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 27 of its 33 references that are in the map. First posted on arXiv 3 Jun 2021.

Builds on

27 of 33 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2020Longformer: The Long-Document Transformercited by 3,362
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  9. 2018Deep contextualized word representationscited by 2,285
  10. 2018Self-Attention with Relative Position Representationscited by 1,112
  11. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  12. 2018Universal Transformerscited by 613
15 more
  1. 2020LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attentioncited by 326
  2. 2019Entity, Relation, and Event Extraction with Contextualized Span Representationscited by 261
  3. 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 204
  4. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  5. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  6. 2018Phrase-Based & Neural Unsupervised Machine Translationcited by 138
  7. 2019Open Sesame: Getting Inside BERT's Linguistic Knowledgecited by 119
  8. 2018Meta-Learning for Low-Resource Neural Machine Translationcited by 105
  9. 2019Do Attention Heads in BERT Track Syntactic Dependencies?cited by 87
  10. 2020Out of Order: How Important Is The Sequential Order of Words in a Sentence in Natural Language Understanding Tasks?cited by 72
  11. 2020Learning to Encode Position for Transformer with Continuous Dynamical Modelcited by 61
  12. 2020What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encodingcited by 52
  13. 2020Pre-Training Transformers as Energy-Based Cloze Modelscited by 52
  14. 2020jiant: A Software Toolkit for Research on General-Purpose Text Understanding Modelscited by 39
  15. 2020Prosody Learning Mechanism for Speech Synthesis System Without Text Length Limitcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 3; 2023: 8; 2024: 4; 2025: 2; 2026: 3.

23 papers.

Sort:

By area:

  1. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  2. 2022Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Modelscited by 188
  3. 2023Matcha-TTS: A fast TTS architecture with conditional flow matchingcited by 142
  4. 2022KERPLE: Kernelized Relative Positional Embedding for Length Extrapolationcited by 84
  5. 2023Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encodingcited by 32
  6. 2024The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Surveycited by 32
  7. 2023Diffusion-Based Co-Speech Gesture Generation Using Joint Text and Audio Representationcited by 27
  8. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 25
  9. 2021Transflower: probabilistic autoregressive dance generation with multimodal attentioncited by 22
  10. 2023Monotonic Location Attention for Length Generalizationcited by 9
  11. 2022Transformers discover an elementary calculation system exploiting local attention and grid-like problem representationcited by 7
  12. 2023Unified speech and gesture synthesis using flow matchingcited by 6
11 more
  1. 2024Speech-driven Personalized Gesture Synthetics: Harnessing Automatic Fuzzy Feature Inferencecited by 6
  2. 2024Are queries and keys always relevant? A case study on Transformer wave functionscited by 5
  3. 2025Group Representational Position Encodingcited by 5
  4. 2023LEA: Improving Sentence Similarity Robustness to Typos Using Lexical Attention Biascited by 2
  5. 2021Shatter: An Efficient Transformer Encoder with Single-Headed Self-Attention and Relative Sequence Partitioningcited by 1
  6. 2025Order-Level Attention Similarity Across Language Models: A Latent Commonalitycited by 1
  7. 2026Attention's Gravitational Field:A Power-Law Interpretation of Positional Correlationcited by 1
  8. 2023Extending Input Contexts of Language Models through Training on Segmented Sequencescited by 0
  9. 2024Exploring Internal Numeracy in Language Models: A Case Study on ALBERTcited by 0
  10. 2026When Attention Goes Blind: Numerical Failure in ALiBi Positional Encodingscited by 0
  11. 2026Flexible Motion Generation from Language and Style Referencescited by 0