mlsanityData to 7 Oct 2026

KERPLE: Kernelized Relative Positional Embedding for Length Extrapolation

arXiv 2205.09921 ↗First posted 20 May 2022cs.CL cs.LGcited by 84

Not on the map. Why?

Abstract (arXiv)

Relative positional embeddings (RPE) have received considerable attention since RPEs effectively model the relative distance among tokens and enable length extrapolation. We propose KERPLE, a framework that generalizes relative position embedding for extrapolation by kernelizing positional differences. We achieve this goal using conditionally positive definite (CPD) kernels, a class of functions known for generalizing distance metrics. To maintain the inner product interpretation of self-attention, we show that a CPD kernel can be transformed into a PD kernel by adding a constant offset. This offset is implicitly absorbed in the Softmax normalization during self-attention. The diversity of CPD kernels allows us to derive various RPEs that enable length extrapolation in a principled way. Experiments demonstrate that the logarithmic variant achieves excellent extrapolation performance on three large language modeling datasets. Our implementation and pretrained checkpoints are released at https://github.com/chijames/KERPLE.git.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). Builds on 36 of its 57 references that are in the map. First posted on arXiv 20 May 2022.

Builds on

36 of 57 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  9. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  11. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  12. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
24 more
  1. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  3. 2020Rethinking Attention with Performerscited by 1,584
  4. 2018Self-Attention with Relative Position Representationscited by 1,112
  5. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  6. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  7. 2020Recipes for building an open-domain chatbotcited by 593
  8. 2017Convolutional Sequence to Sequence Learningcited by 550
  9. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  10. 2021Random Feature Attentioncited by 350
  11. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 227
  12. 2022cosFormer: Rethinking Softmax in Attentioncited by 212
  13. 2019Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernelcited by 211
  14. 2021Rethinking and Improving Relative Position Encoding for Vision Transformercited by 206
  15. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  16. 2021Position Information in Transformers: An Overviewcited by 103
  17. 2020Improve Transformer Models with Better Relative Position Embeddingscited by 73
  18. 2021Skyformer: Remodel Self-Attention with Gaussian Kernel and Nystr"om Methodcited by 61
  19. 2021Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encodingcited by 50
  20. 2021CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddingscited by 47
  21. 2021A Simple and Effective Positional Encoding for Transformerscited by 37
  22. 2021SHAPE: Shifted Absolute Position Embedding for Transformerscited by 37
  23. 2020DA-Transformer: Distance-aware Transformercited by 23
  24. 2021The Case for Translation-Invariant Self-Attention in Transformer-Based Language Modelscited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 14; 2024: 23; 2025: 31; 2026: 14.

84 papers.

Sort:

By area:

  1. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  2. 2022A Length-Extrapolatable Transformercited by 123
  3. 2024Transformers Can Achieve Length Generalization But Not Robustlycited by 82
  4. 2024Transformers Can Do Arithmetic with the Right Embeddingscited by 77
  5. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  6. 2024What is Wrong with Perplexity for Long-context Language Modeling?cited by 68
  7. 2023Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Contextcited by 63
  8. 2024Human-inspired Episodic Memory for Infinite Context LLMscited by 62
  9. 2025On the Emergence of Position Bias in Transformerscited by 57
  10. 2025Forgetting Transformer: Softmax Attention with a Forget Gatecited by 50
  11. 2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 46
  12. 2023CLEX: Continuous Length Extrapolation for Large Language Modelscited by 43
38 more
  1. 2022Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysiscited by 39
  2. 2024Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalizationcited by 37
  3. 2024DAPE: Data-Adaptive Positional Encoding for Length Extrapolationcited by 30
  4. 2023Primal-Attention: Self-attention through Asymmetric Kernel SVD in Primal Representationcited by 28
  5. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 25
  6. 2024Universal Length Generalization with Turing Programscited by 25
  7. 2024Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolationcited by 23
  8. 2024LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Modelscited by 23
  9. 2025Learning the RoPEs: Better 2D and 3D Position Encodings with STRINGcited by 19
  10. 2024Exploring Context Window of Large Language Models via Decomposed Positional Vectorscited by 18
  11. 2025Long-Context Generalization with Sparse Attentioncited by 18
  12. 2023Linearized Relative Positional Encodingcited by 17
  13. 2024HoPE: A Novel Positional Encoding Without Long-Term Decay for Enhanced Context Awareness and Extrapolationcited by 17
  14. 2024BA-SAM: Scalable Bias-Mode Attention Mask for Segment Anything Modelcited by 15
  15. 2024UncertaintyRAG: Span-Level Uncertainty Enhanced Long-Context Modeling for Retrieval-Augmented Generationcited by 14
  16. 2023Transformer Working Memory Enables Regular Language Reasoning and Natural Language Length Extrapolationcited by 13
  17. 2024CLongEval: A Chinese Benchmark for Evaluating Long-Context Large Language Modelscited by 13
  18. 2025ParallelComp: Parallel Long-Context Compressor for Length Extrapolationcited by 13
  19. 2025Native-Resolution Image Synthesiscited by 13
  20. 2024Rethinking Transformer for Long Contextual Histopathology Whole Slide Image Analysiscited by 12
  21. 2025A Survey on Transformer Context Extension: Approaches and Evaluationcited by 11
  22. 2023From Interpolation to Extrapolation: Complete Length Generalization for Arithmetic Transformerscited by 10
  23. 2024DAPE V2: Process Attention Score as Feature Map for Length Extrapolationcited by 10
  24. 2025HoPE: Hyperbolic Rotary Positional Encoding for Stable Long-Range Dependency Modeling in Large Language Modelscited by 10
  25. 2023Attention Alignment and Flexible Positional Embeddings Improve Transformer Length Extrapolationcited by 9
  26. 2023Exploring Transformer Extrapolationcited by 8
  27. 2024Let the Code LLM Edit Itself When You Edit the Codecited by 8
  28. 2025Selective Rotary Position Embeddingcited by 8
  29. 2024How Transformers Get Rich: Approximation and Dynamics Analysiscited by 7
  30. 2025On Vanishing Variance in Transformer Length Generalizationcited by 6
  31. 2023CoCA: Fusing Position Embedding with Collinear Constrained Attention in Transformers for Long Context Window Extendingcited by 5
  32. 2025Group Representational Position Encodingcited by 5
  33. 2024Self-Attention through Kernel-Eigen Pair Sparse Variational Gaussian Processescited by 4
  34. 2025Rethinking Addressing in Language Models via Contexualized Equivariant Positional Encodingcited by 4
  35. 2025Where is this coming from? Making groundedness count in the evaluation of Document VQA modelscited by 4
  36. 2025Deconstructing Positional Information: From Attention Logits to Training Biasescited by 4
  37. 2023Long-MIL: Scaling Long Contextual Multiple Instance Learning for Histopathology Whole Slide Image Analysiscited by 3
  38. 2024MEP: Multiple Kernel Learning Enhancing Relative Positional Encoding Length Extrapolationcited by 3

50 most cited of 84 All 84 (JSON)