mlsanityData to 7 Oct 2026

3D-RPE: Enhancing Long-Context Modeling Through 3D Rotary Position Encoding

arXiv 2406.09897 ↗First posted 14 Jun 2024cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

Inspired by the Bloch Sphere representation, we propose a novel rotary position encoding on a three-dimensional sphere, named 3D Rotary Position Encoding (3D-RPE). 3D-RPE is an advanced version of the widely used 2D Rotary Position Encoding (RoPE), with two major advantages for modeling long contexts: controllable long-term decay and improved position resolution. For controllable long-term decay, 3D-RPE allows for the regulation of long-term decay within the chunk size, ensuring the modeling of relative positional information between tokens at a distant relative position. For enhanced position resolution, 3D-RPE can mitigate the degradation of position resolution caused by position interpolation on RoPE. We have conducted experiments on long-context Natural Language Understanding (NLU) and long-sequence Language Modeling (LM) tasks. From the experimental results, 3D-RPE achieved performance improvements over RoPE, especially in long-context NLU tasks.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 21 of its 29 references that are in the map. First posted on arXiv 14 Jun 2024.

Builds on

21 of 29 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  9. 2018Self-Attention with Relative Position Representationscited by 1,112
  10. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  11. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  12. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
9 more
  1. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  2. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  3. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  4. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  5. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  6. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  7. 2024Long Context Compression with Activation Beaconcited by 101
  8. 2024Training-Free Long-Context Scaling of Large Language Modelscited by 61
  9. 2019Encoding word order in complex embeddingscited by 53

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 2; 2026: 8.

13 papers.

Sort:

By area:

  1. 2024Eliminating Position Bias of Language Models: A Mechanistic Approachcited by 61
  2. 2024Why Does the Effective Context Length of LLMs Fall Short?cited by 60
  3. 2024DAPE V2: Process Attention Score as Feature Map for Length Extrapolationcited by 10
  4. 2026SceneTok: A Compressed, Diffusable Token Space for 3D Scenescited by 6
  5. 2026SigVLP: Sigmoid Volume-Language Pre-Training for Self-Supervised CT-Volume Adaptive Representation Learningcited by 2
  6. 2026nD-RoPE: A Generalized RoPE for n-Dimensional Position Embeddingcited by 2
  7. 2025CCF: A Context Compression Framework for Efficient Long-Sequence Language Modelingcited by 1
  8. 2025OMEGA: Optimized Multimodal Position Encoding Index Derivation with Global Adaptive Scaling for Vision-Language Modelscited by 1
  9. 2026VIRST: Video-Instructed Reasoning Assistant for SpatioTemporal Segmentationcited by 1
  10. 2026CryoACE: An Atom-centric Framework for Accurate and Automated Model Building in Cryo-EMcited by 1
  11. 2026LACE: Lattice Attention for Cross-thread Explorationcited by 0
  12. 2026Higher-Dimensional Rotary Position Embeddingcited by 0
1 more
  1. 2026Line-Coupled Language Modelcited by 0