mlsanityData to 7 Oct 2026

Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysis

arXiv 2212.10356 ↗First posted 20 Dec 2022cs.CLcited by 39

Not on the map. Why?

Abstract (arXiv)

Length extrapolation permits training a transformer language model on short sequences that preserves perplexities when tested on substantially longer sequences. A relative positional embedding design, ALiBi, has had the widest usage to date. We dissect ALiBi via the lens of receptive field analysis empowered by a novel cumulative normalized gradient tool. The concept of receptive field further allows us to modify the vanilla Sinusoidal positional embedding to create ~Sandwich, the first parameter-free relative positional embedding design that truly length information uses longer than the training sequence. Sandwich shares with KERPLE and T5 the same logarithmic decaying temporal bias pattern with learnable relative positional embeddings; these elucidate future extrapolatable positional embedding design.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 20 of its 27 references that are in the map. First posted on arXiv 20 Dec 2022.

Builds on

20 of 27 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2016Layer Normalizationcited by 5,579
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  7. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  8. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  9. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  10. 2017Deformable Convolutional Networkscited by 1,481
  11. 2015Multi-Scale Context Aggregation by Dilated Convolutionscited by 1,128
  12. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
8 more
  1. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  2. 2017Understanding the Effective Receptive Field in Deep Convolutional Neural Networkscited by 549
  3. 2022Exploring Length Generalization in Large Language Modelscited by 204
  4. 2014Recurrent Neural Network Regularizationcited by 189
  5. 2022KERPLE: Kernelized Relative Positional Embedding for Length Extrapolationcited by 84
  6. 2017Dilated Recurrent Neural Networkscited by 83
  7. 2019TENER: Adapting Transformer Encoder for Named Entity Recognitioncited by 69
  8. 2020GMAT: Global Memory Augmentation for Transformerscited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 13; 2025: 12; 2026: 2.

39 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  3. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  4. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 120
  5. 2024Transformers Can Do Arithmetic with the Right Embeddingscited by 77
  6. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  7. 2024Training-Free Long-Context Scaling of Large Language Modelscited by 61
  8. 2024Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Modelscited by 57
  9. 2025Forgetting Transformer: Softmax Attention with a Forget Gatecited by 50
  10. 2024Base of RoPE Bounds Context Lengthcited by 49
  11. 2024DeciMamba: Exploring the Length Extrapolation Potential of Mambacited by 45
  12. 2023CLEX: Continuous Length Extrapolation for Large Language Modelscited by 43
27 more
  1. 2023Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encodingcited by 32
  2. 2024DAPE: Data-Adaptive Positional Encoding for Length Extrapolationcited by 30
  3. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 25
  4. 2024Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolationcited by 23
  5. 2025Learning the RoPEs: Better 2D and 3D Position Encodings with STRINGcited by 19
  6. 2024Exploring Context Window of Large Language Models via Decomposed Positional Vectorscited by 18
  7. 2024HoPE: A Novel Positional Encoding Without Long-Term Decay for Enhanced Context Awareness and Extrapolationcited by 17
  8. 2024Towards LifeSpan Cognitive Systemscited by 13
  9. 2025A Survey on Transformer Context Extension: Approaches and Evaluationcited by 11
  10. 2023From Interpolation to Extrapolation: Complete Length Generalization for Arithmetic Transformerscited by 10
  11. 2024DAPE V2: Process Attention Score as Feature Map for Length Extrapolationcited by 10
  12. 2025HoPE: Hyperbolic Rotary Positional Encoding for Stable Long-Range Dependency Modeling in Large Language Modelscited by 10
  13. 2023Attention Alignment and Flexible Positional Embeddings Improve Transformer Length Extrapolationcited by 9
  14. 2024Let the Code LLM Edit Itself When You Edit the Codecited by 8
  15. 2025Position Bias Mitigates Position Bias:Mitigate Position Bias Through Inter-Position Knowledge Distillationcited by 8
  16. 2023Zebra: Extending Context Window with Layerwise Grouped Local-Global Attentioncited by 6
  17. 2025Group Representational Position Encodingcited by 5
  18. 2025Rethinking Addressing in Language Models via Contexualized Equivariant Positional Encodingcited by 4
  19. 2025Deconstructing Positional Information: From Attention Logits to Training Biasescited by 4
  20. 2024MEP: Multiple Kernel Learning Enhancing Relative Positional Encoding Length Extrapolationcited by 3
  21. 2025Long-Short Alignment for Effective Long-Context Modeling in LLMscited by 3
  22. 2023LMTuner: An user-friendly and highly-integrable Training Framework for fine-tuning Large Language Modelscited by 2
  23. 2025Context-aware Biases for Length Extrapolationcited by 2
  24. 2025A Circular Argument : Does RoPE need to be Equivariant for Vision?cited by 2
  25. 2025TokenSmith: Streamlining Data Editing, Search, and Inspection for Large-Scale Language Model Training and Interpretabilitycited by 0
  26. 2026Give it Space! Explicit Disentangling of Positional and Semantic Representations in Encoderscited by 0
  27. 2026When Attention Goes Blind: Numerical Failure in ALiBi Positional Encodingscited by 0