mlsanityData to 7 Oct 2026

Why Are Positional Encodings Nonessential for Deep Autoregressive Transformers? Revisiting a Petroglyph

arXiv 2501.00659 ↗First posted 31 Dec 2024cs.LG cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Do autoregressive Transformer language models require explicit positional encodings (PEs)? The answer is 'no' provided they have more than one layer -- they can distinguish sequences with permuted tokens without the need for explicit PEs. This follows from the fact that a cascade of (permutation invariant) set processors can collectively exhibit sequence-sensitive behavior in the autoregressive setting. This property has been known since early efforts (contemporary with GPT-2) adopting the Transformer for language modeling. However, this result does not appear to have been well disseminated, leading to recent rediscoveries. This may be partially due to a sudden growth of the language modeling community after the advent of GPT-2/3, but perhaps also due to the lack of a clear explanation in prior work, despite being commonly understood by practitioners in the past. Here we review the long-forgotten explanation why explicit PEs are nonessential for multi-layer autoregressive Transformers (in contrast, one-layer models require PEs to discern order information of their inputs), as well as the origin of this result, and hope to re-establish it as a common knowledge.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 43 of its 66 references that are in the map. First posted on arXiv 31 Dec 2024.

Builds on

43 of 66 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  8. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  9. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  11. 2018Self-Attention with Relative Position Representationscited by 1,112
  12. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
31 more
  1. 2017Convolutional Sequence to Sequence Learningcited by 550
  2. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  3. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  4. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  5. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  6. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  7. 2017A Structured Self-attentive Sentence Embeddingcited by 318
  8. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  9. 2016A Decomposable Attention Model for Natural Language Inferencecited by 214
  10. 2019Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernelcited by 211
  11. 2014Recurrent Neural Network Regularizationcited by 189
  12. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  13. 2016Long Short-Term Memory-Networks for Machine Readingcited by 161
  14. 2016Using Fast Weights to Attend to the Recent Pastcited by 161
  15. 2022Transformer Language Models without Positional Encodings Still Learn Positional Informationcited by 148
  16. 2018Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNNcited by 142
  17. 2018Character-Level Language Modeling with Deeper Self-Attentioncited by 138
  18. 2021The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformerscited by 117
  19. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLMcited by 106
  20. 2019Language Modeling with Deep Transformerscited by 96
  21. 2022What Language Model to Train if You Have One Million GPU Hours?cited by 92
  22. 2016Quasi-Recurrent Neural Networkscited by 87
  23. 2021Going Beyond Linear Transformers with Recurrent Fast Weight Programmerscited by 84
  24. 2017DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understandingcited by 71
  25. 2021The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalizationcited by 62
  26. 2017Simple Recurrent Units for Highly Parallelizable Recurrencecited by 60
  27. 2022The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attentioncited by 45
  28. 2013Dropout improves Recurrent Neural Networks for Handwriting Recognitioncited by 39
  29. 2024A phase transition between positional and semantic learning in a solvable model of dot-product attentioncited by 30
  30. 2023Practical Computational Power of Linear Transformers and Their Recurrent and Self-Referential Extensionscited by 20
  31. 2016Strongly-Typed Recurrent Neural Networkscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 1.

6 papers.

  1. 2025Blending Complementary Memory Systems in Hybrid Quadratic-Linear Transformerscited by 12
  2. 2025Decoupling the "What" and "Where" With Polar Coordinate Positional Embeddingscited by 10
  3. 2025Efficient Autoregressive Inference for Transformer Probabilistic Modelscited by 4
  4. 2025Language Models, Graph Searching, and Supervision Adulteration: When More Supervision is Less and How to Make More Morecited by 3
  5. 2025TREASURE: The Visa Payment Foundation Model for High-Volume Transaction Understandingcited by 3
  6. 2026Position Encoding with Random Float Sampling Enhances Length Generalization of Transformerscited by 0