mlsanityData to 14 Oct 2026

Exploring Transformer Extrapolation

arXiv 2307.10156 ↗First posted 19 Jul 2023cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

Length extrapolation has attracted considerable attention recently since it allows transformers to be tested on longer sequences than those used in training. Previous research has shown that this property can be attained by using carefully designed Relative Positional Encodings (RPEs). While these methods perform well on a variety of corpora, the conditions for length extrapolation have yet to be investigated. This paper attempts to determine what types of RPEs allow for length extrapolation through a thorough mathematical and empirical analysis. We discover that a transformer is certain to possess this property as long as the series that corresponds to the RPE's exponential converges. Two practices are derived from the conditions and examined in language modeling tasks on a variety of corpora. As a bonus from the conditions, we derive a new Theoretical Receptive Field (TRF) to measure the receptive field of RPEs without taking any training steps. Extensive experiments are conducted on the Wikitext-103, Books, Github, and WikiBook datasets to demonstrate the viability of our discovered conditions. We also compare TRF to Empirical Receptive Field (ERF) across different models, showing consistently matched trends on the aforementioned datasets. The code is available at https://github.com/OpenNLPLab/Rpe.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 31 of its 39 references that are in the map. First posted on arXiv 19 Jul 2023.

Builds on

31 of 39 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  6. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  7. 2020Longformer: The Long-Document Transformercited by 3,413
  8. 2016Pointer Sentinel Mixture Modelscited by 3,096
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  10. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  11. 2021ViViT: A Video Vision Transformercited by 1,620
  12. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
19 more
  1. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,127
  2. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  3. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  4. 2016CNN Architectures for Large-Scale Audio Classificationcited by 704
  5. 2021AST: Audio Spectrogram Transformercited by 643
  6. 2017Convolutional Sequence to Sequence Learningcited by 552
  7. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
  8. 2022cosFormer: Rethinking Softmax in Attentioncited by 213
  9. 2022Should You Mask 15% in Masked Language Modeling?cited by 118
  10. 2022The Devil in Linear Transformercited by 98
  11. 2022KERPLE: Kernelized Relative Positional Embedding for Length Extrapolationcited by 86
  12. 2021Skyformer: Remodel Self-Attention with Gaussian Kernel and Nystr"om Methodcited by 63
  13. 2023Toeplitz Neural Network for Sequence Modelingcited by 43
  14. 2021Relative Positional Encoding for Transformers with Linear Complexitycited by 37
  15. 2022Vicinity Vision Transformercited by 22
  16. 2022Neural Architecture Search on Efficient Transformers and Beyondcited by 17
  17. 2023Linearized Relative Positional Encodingcited by 17
  18. 2022Locality Matters: A Locality-Biased Linear Attention for Automatic Speech Recognitioncited by 6
  19. 2022Linear Video Transformer with Feature Fixationcited by 6

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 6; 2025: 1.

8 papers.

  1. 2024Titans: Learning to Memorize at Test Timecited by 328
  2. 2024Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Modelscited by 57
  3. 2024DAPE: Data-Adaptive Positional Encoding for Length Extrapolationcited by 31
  4. 2024Various Lengths, Constant Speed: Efficient Language Modeling with Lightning Attentioncited by 31
  5. 2024DAPE V2: Process Attention Score as Feature Map for Length Extrapolationcited by 10
  6. 2023Navigating Scaling Laws: Compute Optimality in Adaptive Model Trainingcited by 4
  7. 2024MLissard: Multilingual Long and Simple Sequential Reasoning Benchmarkscited by 1
  8. 2025A Vision Toward Energy-Efficient Domain-Specific Artificial Intelligence Models and Agentscited by 0