mlsanityData to 7 Oct 2026

Position Information in Transformers: An Overview

arXiv 2102.11090 ↗First posted 22 Feb 2021cs.CL cs.AIcited by 103

Not on the map. Why?

Abstract (arXiv)

Transformers are arguably the main workhorse in recent Natural Language Processing research. By definition a Transformer is invariant with respect to reordering of the input. However, language is inherently sequential and word order is essential to the semantics and syntax of an utterance. In this article, we provide an overview and theoretical comparison of existing methods to incorporate position information into Transformer models. The objectives of this survey are to (1) showcase that position information in Transformer is a vibrant and extensive research area; (2) enable the reader to compare existing methods by providing a unified notation and systematization of different approaches along important model dimensions; (3) indicate what characteristics of an application should be taken into account when selecting a position encoding; (4) provide stimuli for future research.

Key facts. Cited by 103 papers in the map (data through 7 Oct 2026). Builds on 45 of its 67 references that are in the map. First posted on arXiv 22 Feb 2021.

Builds on

45 of 67 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2016Layer Normalizationcited by 5,579
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  9. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  10. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  11. 2020Longformer: The Long-Document Transformercited by 3,362
  12. 2018Deep contextualized word representationscited by 2,285
33 more
  1. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  2. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  3. 2020Reformer: The Efficient Transformercited by 1,658
  4. 2020Rethinking Attention with Performerscited by 1,584
  5. 2018Self-Attention with Relative Position Representationscited by 1,112
  6. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  7. 2020A Generalization of Transformer Networks to Graphscited by 653
  8. 2018Universal Transformerscited by 613
  9. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  10. 2017Convolutional Sequence to Sequence Learningcited by 550
  11. 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 204
  12. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  13. 2020Graph-Bert: Only Attention is Needed for Learning Graph Representationscited by 176
  14. 2014A Convolutional Neural Network for Modelling Sentencescited by 163
  15. 2019Graph Transformer for Graph-to-Sequence Learningcited by 82
  16. 2020Shortformer: Better Language Modeling using Shorter Inputscited by 76
  17. 2020Improve Transformer Models with Better Relative Position Embeddingscited by 73
  18. 2017DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understandingcited by 71
  19. 2019TENER: Adapting Transformer Encoder for Named Entity Recognitioncited by 69
  20. 2020Learning to Encode Position for Transformer with Continuous Dynamical Modelcited by 61
  21. 2019Positional Encoding to Control Output Sequence Lengthcited by 56
  22. 2016Fully Character-Level Neural Machine Translation without Explicit Segmentationcited by 55
  23. 2019Encoding word order in complex embeddingscited by 53
  24. 2020What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encodingcited by 52
  25. 2020Hard-Coded Gaussian Attention for Neural Machine Translationcited by 42
  26. 2019Modeling Graph Structure in Transformer for Better AMR-to-Text Generationcited by 35
  27. 2020Improving Zero-Shot Translation by Disentangling Positional Informationcited by 35
  28. 2020Self-Attention with Cross-Lingual Position Representationcited by 27
  29. 2020DA-Transformer: Distance-aware Transformercited by 23
  30. 2019Self-Attention with Structural Position Representationscited by 21
  31. 2019Non-autoregressive Transformer by Position Learningcited by 17
  32. 2019Assessing the Ability of Self-Attention Networks to Learn Word Ordercited by 9
  33. 2019An Augmented Transformer Architecture for Natural Language Generation Taskscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 14; 2023: 21; 2024: 27; 2025: 21; 2026: 11.

103 papers.

Sort:

By area:

  1. 2022Multimodal Learning with Transformers: A Surveycited by 399
  2. 2021Graph Neural Networks with Learnable Structural and Positional Representationscited by 326
  3. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  4. 2022A Length-Extrapolatable Transformercited by 123
  5. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  6. 2022KERPLE: Kernelized Relative Positional Embedding for Length Extrapolationcited by 84
  7. 2022CRAFT: Cross-Attentional Flow Transformer for Robust Optical Flowcited by 82
  8. 2024Contextual Position Encoding: Learning to Count What's Importantcited by 53
  9. 2023Invariant Slot Attention: Object Discovery with Slot-Centric Reference Framescited by 51
  10. 2021CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddingscited by 47
  11. 2021Molformer: Motif-based Transformer on 3D Heterogeneous Molecular Graphscited by 44
  12. 2023Improving Position Encoding of Transformers for Multivariate Time Series Classificationcited by 44
38 more
  1. 2024Learning with 3D rotations, a hitchhiker's guide to SO(3)cited by 39
  2. 2021SHAPE: Shifted Absolute Position Embedding for Transformerscited by 37
  3. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  4. 2022Masked Jigsaw Puzzle: A Versatile Position Embedding for Vision Transformerscited by 33
  5. 2023Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encodingcited by 32
  6. 2025PaTH Attention: Position Encoding via Accumulating Householder Transformationscited by 32
  7. 2022Position Prediction as an Effective Pretraining Strategycited by 24
  8. 2024Materials science in the era of large language models: a perspectivecited by 23
  9. 2024Explainable Generative AI (GenXAI): A Survey, Conceptualization, and Research Agendacited by 22
  10. 2023Universal Morphology Control via Contextual Modulationcited by 21
  11. 2023DropPos: Pre-Training Vision Transformers by Reconstructing Dropped Positionscited by 19
  12. 2023It's Just a Matter of Time: Detecting Depression with Time-Enriched Multimodal Transformerscited by 18
  13. 2025Beyond Sensor Data: Foundation Models of Behavioral Data from Wearables Improve Health Predictionscited by 18
  14. 2024Positional Knowledge is All You Need: Position-induced Transformer (PiT) for Operator Learningcited by 17
  15. 2024Code Simulation Challenges for Large Language Modelscited by 15
  16. 2021Perspectives and Prospects on Transformer Architecture for Cross-Modal Tasks with Language and Visioncited by 14
  17. 2022Recent Advances in Neural Text Generation: A Task-Agnostic Surveycited by 14
  18. 2023Bag of Tricks for Effective Language Model Pretraining and Downstream Adaptation: A Case Study on GLUEcited by 11
  19. 2023Causal Decision Transformer for Recommender Systems via Offline Reinforcement Learningcited by 11
  20. 2021CoSformer: Detecting Co-Salient Object with Transformerscited by 10
  21. 2021Shaking Syntactic Trees on the Sesame Street: Multilingual Probing with Controllable Perturbationscited by 10
  22. 2023From Interpolation to Extrapolation: Complete Length Generalization for Arithmetic Transformerscited by 10
  23. 2026The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?cited by 10
  24. 2023Monotonic Location Attention for Length Generalizationcited by 9
  25. 2023Graph Neural Networks with Diverse Spectral Filteringcited by 9
  26. 2023Algebraic Positional Encodingscited by 9
  27. 2025Softmax Transformers are Turing-Completecited by 9
  28. 2025Stronger Enforcement of Instruction Hierarchy via Augmented Intermediate Representationscited by 7
  29. 2023PAT: Position-Aware Transformer for Dense Multi-Label Action Detectioncited by 6
  30. 2024Graph Language Modelscited by 6
  31. 2025Code Simulation as a Proxy for High-order Tasks in Large Language Modelscited by 6
  32. 2025Revisiting LRP: Positional Attribution as the Missing Ingredient for Transformer Explainabilitycited by 6
  33. 2021The Impact of Positional Encodings on Multilingual Compressioncited by 5
  34. 2023Transformers as Graph-to-Graph Modelscited by 5
  35. 2024Knowledge of Pretrained Language Models on Surface Information of Tokenscited by 5
  36. 2024Reference Neural Operators: Learning the Smooth Dependence of Solutions of PDEs on Geometric Deformationscited by 5
  37. 2024Transformer Encoder Satisfiability: Complexity and Impact on Formal Reasoningcited by 5
  38. 2024Shared Imagination: LLMs Hallucinate Alikecited by 5

50 most cited of 103 All 103 (JSON)