mlsanityData to 7 Oct 2026

Position Interpolation Improves ALiBi Extrapolation

arXiv 2310.13017 ↗First posted 18 Oct 2023cs.CL cs.AI cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Linear position interpolation helps pre-trained models using rotary position embeddings (RoPE) to extrapolate to longer sequence lengths. We propose using linear position interpolation to extend the extrapolation range of models using Attention with Linear Biases (ALiBi). We find position interpolation significantly improves extrapolation capability on upstream language modelling and downstream summarization and retrieval tasks.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 6 of its 12 references that are in the map. First posted on arXiv 18 Oct 2023.

Builds on

6 of 12 references.

References · 6

  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  2. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  3. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  4. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  5. 2021Efficient Attentions for Long Document Summarizationcited by 342
  6. 2023BTLM-3B-8K: 7B Parameter Performance in a 3B Parameter Modelcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 1; 2026: 1.

6 papers.

  1. 2024Poro 34B and the Blessing of Multilingualitycited by 18
  2. 2024Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generationcited by 15
  3. 2024Flash STU: Fast Spectral Transform Unitscited by 6
  4. 2024LongSSM: On the Length Extension of State-space Models in Language Modellingcited by 3
  5. 2025Spline-based Transformerscited by 2
  6. 2026What DINO saw: ALiBi positional encoding reduces positional bias in Vision Transformerscited by 1