mlsanityData to 7 Oct 2026

Transformer Language Models without Positional Encodings Still Learn Positional Information

arXiv 2203.16634 ↗First posted 30 Mar 2022cs.CL cs.AI cs.LGcited by 148

Not on the map. Why?

Abstract (arXiv)

Causal transformer language models (LMs), such as GPT-3, typically require some form of positional encoding, such as positional embeddings. However, we show that LMs without any explicit positional encoding are still competitive with standard models, and that this phenomenon is robust across different datasets, model sizes, and sequence lengths. Probing experiments reveal that such models acquire an implicit notion of absolute positions throughout the network, effectively compensating for the missing information. We conjecture that causal attention enables the model to infer the number of predecessors that each token can attend to, thereby approximating its absolute position. Our findings indicate that causal LMs might derive positional awareness not only from the explicit positioning mechanism, but also from the effects of the causal mask.

Key facts. Cited by 148 papers in the map (data through 7 Oct 2026). Builds on 18 of its 26 references that are in the map. First posted on arXiv 30 Mar 2022.

Builds on

18 of 26 references.

References · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  8. 2016Pointer Sentinel Mixture Modelscited by 2,992
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  10. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  11. 2018Self-Attention with Relative Position Representationscited by 1,112
  12. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
6 more
  1. 2017Convolutional Sequence to Sequence Learningcited by 550
  2. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  3. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  4. 2019The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectivescited by 139
  5. 2019Language Modeling with Deep Transformerscited by 96
  6. 2019Improving Transformer Models by Reordering their Sublayerscited by 52

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 13; 2023: 22; 2024: 41; 2025: 50; 2026: 22.

148 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  3. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  4. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  5. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  6. 2024SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformerscited by 331
  7. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  8. 2022Transformers Learn Shortcuts to Automatacited by 260
  9. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  10. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  11. 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 111
  12. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 109
38 more
  1. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  2. 2023Neurons in Large Language Models: Dead, N-gram, Positionalcited by 88
  3. 2024Transformers Can Achieve Length Generalization But Not Robustlycited by 82
  4. 2023In-Context Learning through the Bayesian Prismcited by 78
  5. 2023Exposing Attention Glitches with Flip-Flop Language Modelingcited by 74
  6. 2024Contextual Position Encoding: Learning to Count What's Importantcited by 53
  7. 2024Length Generalization of Causal Transformers without Position Encodingcited by 46
  8. 2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 46
  9. 2025Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understandingcited by 44
  10. 2024A Theoretical Understanding of Self-Correction through In-context Alignmentcited by 43
  11. 2024Number Cookbook: Number Understanding of Language Models and How to Improve Itcited by 41
  12. 2025Weight-sparse transformers have interpretable circuitscited by 40
  13. 2024Mechanistic Understanding and Mitigation of Language Model Non-Factual Hallucinationscited by 38
  14. 2023Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encodingcited by 32
  15. 2024The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Surveycited by 32
  16. 2024Graph Transformers: A Surveycited by 32
  17. 2024In-Context Fine-Tuning for Time-Series Foundation Modelscited by 31
  18. 2024A phase transition between positional and semantic learning in a solvable model of dot-product attentioncited by 30
  19. 2024DAPE: Data-Adaptive Positional Encoding for Length Extrapolationcited by 30
  20. 2022Multitrack Music Transformercited by 28
  21. 2024V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encodingcited by 28
  22. 2024Circuit Complexity Bounds for RoPE-based Transformer Architecturecited by 26
  23. 2024Language Models Need Inductive Biases to Count Inductivelycited by 25
  24. 2024Universal Length Generalization with Turing Programscited by 25
  25. 2023Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddingscited by 23
  26. 2024Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolationcited by 23
  27. 2025Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddingscited by 23
  28. 2022Language model acceptability judgements are not always robust to contextcited by 22
  29. 2024Transformers Provably Learn Sparse Token Selection While Fully-Connected Nets Cannotcited by 21
  30. 2022Systematic Generalization and Emergent Structures in Transformers Trained on Structured Taskscited by 19
  31. 2023Window Attention is Bugged: How not to Interpolate Position Embeddingscited by 18
  32. 2024Exploring Context Window of Large Language Models via Decomposed Positional Vectorscited by 18
  33. 2022The Curious Case of Absolute Position Embeddingscited by 17
  34. 2025SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modelingcited by 16
  35. 2024Scaling Stick-Breaking Attention: An Efficient Implementation and In-depth Studycited by 13
  36. 2025Blending Complementary Memory Systems in Hybrid Quadratic-Linear Transformerscited by 12
  37. 2024DAPE V2: Process Attention Score as Feature Map for Length Extrapolationcited by 10
  38. 2025HoPE: Hyperbolic Rotary Positional Encoding for Stable Long-Range Dependency Modeling in Large Language Modelscited by 10

50 most cited of 148 All 148 (JSON)