mlsanityData to 7 Oct 2026

A Simple and Effective Positional Encoding for Transformers

arXiv 2104.08698 ↗First posted 18 Apr 2021cs.CL cs.LGcited by 37

Not on the map. Why?

Abstract (arXiv)

Transformer models are permutation equivariant. To supply the order and type information of the input tokens, position and segment embeddings are usually added to the input. Recent works proposed variations of positional encodings with relative position encodings achieving better performance. Our analysis shows that the gain actually comes from moving positional information to attention layer from the input. Motivated by this, we introduce Decoupled Positional Attention for Transformers (DIET), a simple yet effective mechanism to encode position and segment information into the Transformer models. The proposed method has faster training and inference time, while achieving competitive performance on GLUE, XTREME and WMT benchmarks. We further generalize our method to long-range transformers and show performance gain.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 22 of its 26 references that are in the map. First posted on arXiv 18 Apr 2021.

Builds on

22 of 26 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  9. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  10. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  11. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  12. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
10 more
  1. 2018Self-Attention with Relative Position Representationscited by 1,112
  2. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  3. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  4. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  5. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
  6. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  7. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 161
  8. 2018Tensor2Tensor for Neural Machine Translationcited by 85
  9. 2020What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encodingcited by 52
  10. 2020Very Deep Transformers for Neural Machine Translationcited by 50

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 6; 2024: 12; 2025: 10; 2026: 5.

37 papers.

Sort:

By area:

  1. 2022KERPLE: Kernelized Relative Positional Embedding for Length Extrapolationcited by 84
  2. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  3. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Modelscited by 71
  4. 2023On the Design Fundamentals of Diffusion Models: A Surveycited by 59
  5. 2024Graph Transformers: A Surveycited by 32
  6. 2022Decoupled Side Information Fusion for Sequential Recommendationcited by 30
  7. 2025Learning the RoPEs: Better 2D and 3D Position Encodings with STRINGcited by 19
  8. 2023Functional Interpolation for Relative Positions Improves Long Context Transformerscited by 18
  9. 2023Knowledge Distillation in Vision Transformers: A Critical Reviewcited by 13
  10. 2024A Review of Transformer-Based Models for Computer Vision Tasks: Capturing Global Context and Spatial Relationshipscited by 11
  11. 2024Enhancing High-order Interaction Awareness in LLM-based Recommender Modelcited by 8
  12. 2025An Empirical Study on Prompt Compression for Large Language Modelscited by 8
25 more
  1. 2022P-Transformer: Towards Better Document-to-Document Neural Machine Translationcited by 5
  2. 2022Word Order Matters when you Increase Maskingcited by 4
  3. 2024Positional encoding is not the same as context: A study on positional encoding for sequential recommendationcited by 4
  4. 2025Survey on Question Answering over Visually Rich Documents: Methods, Challenges, and Trendscited by 4
  5. 2023Long-MIL: Scaling Long Contextual Multiple Instance Learning for Histopathology Whole Slide Image Analysiscited by 3
  6. 2024MEP: Multiple Kernel Learning Enhancing Relative Positional Encoding Length Extrapolationcited by 3
  7. 2024EulerFormer: Sequential User Behavior Modeling with Complex Vector Attentioncited by 3
  8. 2024TeXBLEU: Automatic Metric for Evaluate LaTeX Formatcited by 3
  9. 2023Enhancing Pre-Trained Language Models with Sentence Position Embeddings for Rhetorical Roles Recognition in Legal Opinionscited by 2
  10. 2025LCIRC: A Recurrent Compression Approach for Efficient Long-form Context and Query Dependent Modeling in LLMscited by 2
  11. 2025Context-aware Biases for Length Extrapolationcited by 2
  12. 2025FusionMAE: large-scale pretrained model to optimize and simplify diagnostic and control of fusion plasmacited by 2
  13. 2025What is the Best Sequence Length for BABYLM?cited by 2
  14. 2025GeoPE:A Unified Geometric Positional Embedding for Structured Tensorscited by 2
  15. 2024PCToolkit: A Unified Plug-and-Play Prompt Compression Toolkit of Large Language Modelscited by 1
  16. 2024Transforming NLU with Babylon: A Case Study in Development of Real-time, Edge-Efficient, Multi-Intent Translation System for Automated Drive-Thru Orderingcited by 1
  17. 2024PoPE: Legendre Orthogonal Polynomials Based Position Encoding for Large Language Modelscited by 0
  18. 2024DipMe: Haptic Recognition of Granular Media for Tangible Interactive Applicationscited by 0
  19. 2025Of All StrIPEs: Investigating Structure-informed Positional Encoding for Efficient Music Generationcited by 0
  20. 2025NDLPNet: A Location-Aware Nighttime Deraining Network and a Real-World Benchmark Datasetcited by 0
  21. 2026CRoPE: Efficient Parametrization of Rotary Positional Embeddingcited by 0
  22. 2026An Evaluation of Context Length Extrapolation in Long Code via Positional Embeddings and Efficient Attentioncited by 0
  23. 2026Seeing Beyond: Extrapolative Domain Adaptive Panoramic Segmentationcited by 0
  24. 2026MsFormer: Enabling Robust Predictive Maintenance Services for Industrial Devicescited by 0
  25. 2026Enhancing Transformer-based Routing by Encoding Distance via Relative Positional Encodingcited by 0