mlsanityData to 14 Oct 2026

Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation

arXiv 2401.16421 ↗First posted 29 Jan 2024cs.LG cs.AI cs.CLcited by 24

Not on the map. Why?

Abstract (arXiv)

In this work, we leverage the intrinsic segmentation of language sequences and design a new positional encoding method called Bilevel Positional Encoding (BiPE). For each position, our BiPE blends an intra-segment encoding and an inter-segment encoding. The intra-segment encoding identifies the locations within a segment and helps the model capture the semantic information therein via absolute positional encoding. The inter-segment encoding specifies the segment index, models the relationships between segments, and aims to improve extrapolation capabilities via relative positional encoding. Theoretical analysis shows this disentanglement of positional information makes learning more effective. The empirical results also show that our BiPE has superior length extrapolation capabilities across a wide range of tasks in diverse text modalities.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Builds on 42 of its 57 references that are in the map. First posted on arXiv 29 Jan 2024.

Builds on

42 of 57 references.

References · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2023Let's Verify Step by Stepcited by 3,789
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  11. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  12. 2018Self-Attention with Relative Position Representationscited by 1,120
30 more
  1. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  2. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  3. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  4. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  5. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  6. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 486
  7. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 365
  8. 2021Efficient Attentions for Long Document Summarizationcited by 345
  9. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 327
  10. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 294
  11. 2022Transformers Learn Shortcuts to Automatacited by 268
  12. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 208
  13. 2022Exploring Length Generalization in Large Language Modelscited by 206
  14. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  15. 2022Transformer Language Models without Positional Encodings Still Learn Positional Informationcited by 150
  16. 2021ContractNLI: A Dataset for Document-level Natural Language Inference for Contractscited by 148
  17. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 136
  18. 2022A Length-Extrapolatable Transformercited by 127
  19. 2023Scaling Laws of RoPE-based Extrapolationcited by 125
  20. 2019Fine-tune BERT for Extractive Summarizationcited by 122
  21. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 117
  22. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 111
  23. 2016Hierarchical Multiscale Recurrent Neural Networkscited by 104
  24. 2022KERPLE: Kernelized Relative Positional Embedding for Length Extrapolationcited by 86
  25. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84
  26. 2022Parallel Context Windows for Large Language Modelscited by 81
  27. 2021SummScreen: A Dataset for Abstractive Screenplay Summarizationcited by 79
  28. 2023CLEX: Continuous Length Extrapolation for Large Language Modelscited by 44
  29. 2022Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysiscited by 42
  30. 2023Monotonic Location Attention for Length Generalizationcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 13; 2026: 5.

24 papers.

Sort:

By area:

  1. 2024Training-Free Long-Context Scaling of Large Language Modelscited by 62
  2. 2024SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separatorcited by 41
  3. 2024DAPE: Data-Adaptive Positional Encoding for Length Extrapolationcited by 31
  4. 2025Sample Complexity and Representation Ability of Test-time Scaling Paradigmscited by 12
  5. 2024Let the Code LLM Edit Itself When You Edit the Codecited by 10
  6. 2024DAPE V2: Process Attention Score as Feature Map for Length Extrapolationcited by 10
  7. 2025SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimizationcited by 10
  8. 2025HoPE: Hyperbolic Rotary Positional Encoding for Stable Long-Range Dependency Modeling in Large Language Modelscited by 10
  9. 2025Position Bias Mitigates Position Bias:Mitigate Position Bias Through Inter-Position Knowledge Distillationcited by 9
  10. 2025Born a Transformer -- Always a Transformer? On the Effect of Pretraining on Architectural Abilitiescited by 6
  11. 2025Rethinking Addressing in Language Models via Contexualized Equivariant Positional Encodingcited by 4
  12. 2025SAS: Simulated Attention Scorecited by 4
12 more
  1. 2025Context-aware Biases for Length Extrapolationcited by 2
  2. 2026On the Ability of Transformers to Verify Planscited by 2
  3. 2024Two are better than one: Context window extension with multi-grained self-injectioncited by 1
  4. 2025Softplus Attention with Re-weighting Boosts Length Extrapolation in Large Language Modelscited by 1
  5. 2025LexPro-1.0 Technical Reportcited by 1
  6. 2025Beyond Isolated Capabilities: Bridging Long CoT Reasoning and Long-Context Understandingcited by 1
  7. 2025Position as Probability: Self-Supervised Transformers that Think Past Their Training for Length Extrapolationcited by 0
  8. 2025On the Limitations and Capabilities of Position Embeddings for Length Generalizationcited by 0
  9. 2026Stacked from One: Multi-Scale Self-Injection for Context Window Extensioncited by 0
  10. 2026A Measure-Theoretic Analysis of Reasoning: Structural Generalization and Approximation Limitscited by 0
  11. 2026Give it Space! Explicit Disentangling of Positional and Semantic Representations in Encoderscited by 0
  12. 2026Length Generalization Needs Proper Regularizationcited by 0