mlsanityData to 14 Oct 2026

On the token distance modeling ability of higher RoPE attention dimension

arXiv 2410.08703 ↗First posted 11 Oct 2024cs.CL cs.AIcited by 13

Not on the map. Why?

Abstract (arXiv)

Length extrapolation algorithms based on Rotary position embedding (RoPE) have shown promising results in extending the context length of language models. However, understanding how position embedding can capture longer-range contextual information remains elusive. Based on the intuition that different dimensions correspond to different frequency of changes in RoPE encoding, we conducted a dimension-level analysis to investigate the correlation between a hidden dimension of an attention head and its contribution to capturing long-distance dependencies. Using our correlation metric, we identified a particular type of attention heads, which we named Positional Heads, from various length-extrapolated models. These heads exhibit a strong focus on long-range information interaction and play a pivotal role in long input processing, as evidence by our ablation. We further demonstrate the correlation between the efficiency of length extrapolation and the extension of the high-dimensional attention allocation of these heads. The identification of Positional Heads provides insights for future research in long-text comprehension.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 19 of its 29 references that are in the map. First posted on arXiv 11 Oct 2024.

Builds on

19 of 29 references.

References · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  9. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  10. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  11. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 401
  12. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
7 more
  1. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 232
  2. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 213
  3. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  4. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 186
  5. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 136
  6. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
  7. 2024Base of RoPE Bounds Context Lengthcited by 52

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 7; 2026: 3.

13 papers.

Sort:

By area:

  1. 2024Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMscited by 157
  2. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 50
  3. 2024Feather the Throttle: Revisiting Visual Token Pruning for Vision-Language Model Accelerationcited by 41
  4. 2024Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalizationcited by 39
  5. 2026TriAttention: Efficient Long Reasoning with Trigonometric KV Compressioncited by 21
  6. 2025PhysioWave: A Multi-Scale Wavelet-Transformer for Physiological Signal Representationcited by 12
  7. 2025LeanK: Learnable K Cache Channel Pruning for Efficient Decodingcited by 7
  8. 2025Unveiling Simplicities of Attention: Adaptive Long-Context Head Identificationcited by 5
  9. 2025Attention Debiasing for Token Pruning in Vision Language Modelscited by 4
  10. 2026Modern Transformers Are Implicit Hybrids: From Functional Differentiation to Principled Hybrid Architecture Designcited by 3
  11. 2025Towards Long-window Anchoring in Vision-Language Model Distillationcited by 2
  12. 2025Effective Length Extrapolation via Dimension-Wise Positional Embeddings Manipulationcited by 0
1 more
  1. 2026A Mesoscopic View of Transformer Weights Through Row and Column Scale Fieldscited by 0