mlsanityData to 7 Oct 2026

SHAPE: Shifted Absolute Position Embedding for Transformers

arXiv 2109.05644 ↗First posted 13 Sep 2021cs.CLcited by 37

Not on the map. Why?

Abstract (arXiv)

Position representation is crucial for building position-aware representations in Transformers. Existing position representations suffer from a lack of generalization to test data with unseen lengths or high computational cost. We investigate shifted absolute position embedding (SHAPE) to address both issues. The basic idea of SHAPE is to achieve shift invariance, which is a key property of recent successful position representations, by randomly shifting absolute positions during training. We demonstrate that SHAPE is empirically comparable to its counterpart while being simpler and faster.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 25 of its 32 references that are in the map. First posted on arXiv 13 Sep 2021.

Builds on

25 of 32 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  5. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  6. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  8. 2020Reformer: The Efficient Transformercited by 1,658
  9. 2020Rethinking Attention with Performerscited by 1,584
  10. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  11. 2018Self-Attention with Relative Position Representationscited by 1,112
  12. 2020Efficient Transformers: A Surveycited by 784
13 more
  1. 2017Convolutional Sequence to Sequence Learningcited by 550
  2. 2017OpenNMT: Open-source Toolkit for Neural Machine Translationcited by 251
  3. 2018Scaling Neural Machine Translationcited by 204
  4. 2019Facebook FAIR's WMT19 News Translation Task Submissioncited by 179
  5. 2020Injecting Numerical Reasoning Skills into Language Modelscited by 133
  6. 2021Position Information in Transformers: An Overviewcited by 103
  7. 2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87
  8. 2020Improve Transformer Models with Better Relative Position Embeddingscited by 73
  9. 2019Positional Encoding to Control Output Sequence Lengthcited by 56
  10. 2020The EOS Decision and Length Extrapolationcited by 46
  11. 2020DA-Transformer: Distance-aware Transformercited by 23
  12. 2017Stronger Baselines for Trustable Results in Neural Machine Translationcited by 12
  13. 2021Using Perturbed Length-aware Positional Encoding for Non-autoregressive Neural Machine Translationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 10; 2024: 11; 2025: 7; 2026: 4.

37 papers.

Sort:

By area:

  1. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  2. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 120
  3. 2022KERPLE: Kernelized Relative Positional Embedding for Length Extrapolationcited by 84
  4. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  5. 2023Length Generalization in Arithmetic Transformerscited by 64
  6. 2023Positional Description Matters for Transformers Arithmeticcited by 56
  7. 2024Base of RoPE Bounds Context Lengthcited by 49
  8. 2022Masked Jigsaw Puzzle: A Versatile Position Embedding for Vision Transformerscited by 33
  9. 2023Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encodingcited by 32
  10. 2024The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Surveycited by 32
  11. 2023Learning to Reason and Memorize with Self-Notescited by 30
  12. 2024DAPE: Data-Adaptive Positional Encoding for Length Extrapolationcited by 30
25 more
  1. 2024V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encodingcited by 28
  2. 2024Language Models Need Inductive Biases to Count Inductivelycited by 25
  3. 2025Learning the RoPEs: Better 2D and 3D Position Encodings with STRINGcited by 19
  4. 2022The Curious Case of Absolute Position Embeddingscited by 17
  5. 2024HoPE: A Novel Positional Encoding Without Long-Term Decay for Enhanced Context Awareness and Extrapolationcited by 17
  6. 2022Continual Transformers: Redundancy-Free Attention for Online Inferencecited by 12
  7. 2024DAPE V2: Process Attention Score as Feature Map for Length Extrapolationcited by 10
  8. 2024Investigating Length Issues in Document-level Machine Translationcited by 10
  9. 2023Transformer Language Models Handle Word Frequency in Prediction Headcited by 8
  10. 2024Positional Encoding Helps Recurrent Neural Networks Handle a Large Vocabularycited by 5
  11. 2025Group Representational Position Encodingcited by 5
  12. 2025Deconstructing Positional Information: From Attention Logits to Training Biasescited by 4
  13. 2023Long-MIL: Scaling Long Contextual Multiple Instance Learning for Histopathology Whole Slide Image Analysiscited by 3
  14. 2025Language Models, Graph Searching, and Supervision Adulteration: When More Supervision is Less and How to Make More Morecited by 3
  15. 2022Rethinking the Zigzag Flattening for Image Readingcited by 2
  16. 2025Context-aware Biases for Length Extrapolationcited by 2
  17. 2025Rotary Position Encodings for Graphscited by 1
  18. 2023Extending Input Contexts of Language Models through Training on Segmented Sequencescited by 0
  19. 2024GridPE: A Grid Cell-Inspired Unified Position Embedding for Arbitrary-Dimensional Spacescited by 0
  20. 2024Rethinking Associative Memory Mechanism in Induction Headcited by 0
  21. 2025Softplus Attention with Re-weighting Boosts Length Extrapolation in Large Language Modelscited by 0
  22. 2026A Unified Masked Jigsaw Puzzle Framework for Vision and Language Modelscited by 0
  23. 2026Position Encoding with Random Float Sampling Enhances Length Generalization of Transformerscited by 0
  24. 2026Give it Space! Explicit Disentangling of Positional and Semantic Representations in Encoderscited by 0
  25. 2026Doc2FRC: Length-Consistent Document-Level Machine Translation via Fixed-Range Chunkingcited by 0