# Position Interpolation Improves ALiBi Extrapolation

arXiv: https://arxiv.org/abs/2310.13017 · HTML: https://mlsanity.com/paper/2310.13017/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 12 references that are in the map.
- First posted on arXiv 18 Oct 2023.

## Builds on (most cited first, 6 of 6)

- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization](https://mlsanity.com/paper/2104.05938/) (2021)
- [Efficient Attentions for Long Document Summarization](https://mlsanity.com/paper/2104.02112/) (2021)
- [BTLM-3B-8K: 7B Parameter Performance in a 3B Parameter Model](https://mlsanity.com/paper/2309.11568/) (2023)

## Built on by (most cited first, 6 of 6)

- [Poro 34B and the Blessing of Multilinguality](https://mlsanity.com/paper/2404.01856/) (2024)
- [Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generation](https://mlsanity.com/paper/2404.06910/) (2024)
- [Flash STU: Fast Spectral Transform Units](https://mlsanity.com/paper/2409.10489/) (2024)
- [LongSSM: On the Length Extension of State-space Models in Language Modelling](https://mlsanity.com/paper/2406.02080/) (2024)
- [Spline-based Transformers](https://mlsanity.com/paper/2504.02797/) (2025)
- [What DINO saw: ALiBi positional encoding reduces positional bias in Vision Transformers](https://mlsanity.com/paper/2603.16840/) (2026)

## Abstract (arXiv)

Linear position interpolation helps pre-trained models using rotary position embeddings (RoPE) to extrapolate to longer sequence lengths. We propose using linear position interpolation to extend the extrapolation range of models using Attention with Linear Biases (ALiBi). We find position interpolation significantly improves extrapolation capability on upstream language modelling and downstream summarization and retrieval tasks.