mlsanityData to 7 Oct 2026

Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalization

arXiv 2412.17739 ↗First posted 23 Dec 2024cs.AI cs.CLcited by 37

Not on the map. Why?

Abstract (arXiv)

Extending the context length of Language Models (LMs) by improving Rotary Position Embedding (RoPE) has become a trend. While prior works mainly address RoPE's limitations within attention, this paper uncovers the adverse effects on length generalization from nearly all parts of LMs. Using Discrete Signal Processing theory, we show that RoPE enables periodic attention by implicitly achieving Non-Uniform Discrete Fourier Transform. However, this periodicity is undermined by the spectrum damage caused by: 1) linear layers and activation functions; 2) insufficiently trained frequency components brought by time-domain truncation. Building on our observations, we propose Fourier Position Embedding (FoPE), which enhances attention's frequency-domain properties to improve both its periodic extension and length generalization. FoPE constructs textitFourier Series and zero-outs the destructive frequency components, increasing model robustness against the spectrum damage. Experiments across various model scales and benchmarks show that, within varying context windows, FoPE maintains a more stable performance compared to other baselines. Several analyses and ablations bring further support to our method and theoretical modeling.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 37 of its 44 references that are in the map. First posted on arXiv 23 Dec 2024.

Builds on

37 of 44 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  7. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  8. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  10. 2018Self-Attention with Relative Position Representationscited by 1,112
  11. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  12. 2024OLMo: Accelerating the Science of Language Modelscited by 599
25 more
  1. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  2. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  3. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  4. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  5. 2021Efficient Attentions for Long Document Summarizationcited by 342
  6. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  7. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  8. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  9. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 133
  10. 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 111
  11. 2024CycleNet: Enhancing Time Series Forecasting through Modeling Periodic Patternscited by 102
  12. 2023Neurons in Large Language Models: Dead, N-gram, Positionalcited by 88
  13. 2022KERPLE: Kernelized Relative Positional Embedding for Length Extrapolationcited by 84
  14. 2019Encoding word order in complex embeddingscited by 53
  15. 2024Length Generalization of Causal Transformers without Position Encodingcited by 46
  16. 2023CLEX: Continuous Length Extrapolation for Large Language Modelscited by 43
  17. 2020Language Through a Prism: A Spectral Approach for Multiscale Language Representationscited by 39
  18. 2024Exploring Context Window of Large Language Models via Decomposed Positional Vectorscited by 18
  19. 2024Mixture of In-Context Experts Enhance LLMs' Long Context Awarenesscited by 17
  20. 2024HoPE: A Novel Positional Encoding Without Long-Term Decay for Enhanced Context Awareness and Extrapolationcited by 17
  21. 2019Fourier Neural Networks: A Comparative Studycited by 14
  22. 2024On the token distance modeling ability of higher RoPE attention dimensioncited by 13
  23. 2023Learning a Fourier Transform for Linear Relative Positional Encodings in Transformerscited by 10
  24. 2024Fourier Head: Helping Large Language Models Learn Complex Probability Distributionscited by 10
  25. 2024SMR: State Memory Replay for Long Sequence Modelingcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 16; 2026: 21.

37 papers.

Sort:

By area:

  1. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
  2. 2026Intern-S1-Pro: Scientific Multimodal Foundation Model at Trillion Scalecited by 25
  3. 2025Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddingscited by 23
  4. 2025FoNE: Precise Single-Token Number Embeddings via Fourier Featurescited by 21
  5. 2026Addressable Memory for Video World Modelscited by 11
  6. 2026Convergent Evolution: How Different Language Models Learn Similar Number Representationscited by 5
  7. 2025EnergyFormer: Energy Attention with Fourier Embedding for Hyperspectral Image Classificationcited by 4
  8. 2025Deconstructing Positional Information: From Attention Logits to Training Biasescited by 4
  9. 2025CodeSSM: Towards State Space Models for Code Understandingcited by 3
  10. 2026MrRoPE: Mixed-radix Rotary Position Embeddingcited by 3
  11. 2025SeqPE: Transformer with Sequential Position Encodingcited by 2
  12. 2025Beyond Real: Imaginary Extension of Rotary Position Embeddings for Long-Context LLMscited by 2
25 more
  1. 2025Towards Long-window Anchoring in Vision-Language Model Distillationcited by 2
  2. 2026Attention Needs to Focus: A Unified Perspective on Attention Allocationcited by 2
  3. 2026SoPE: Spherical Coordinate-Based Positional Embedding for Enhancing Spatial Perception of 3D LVLMscited by 2
  4. 2026nD-RoPE: A Generalized RoPE for n-Dimensional Position Embeddingcited by 2
  5. 2025CoPE: A Lightweight Complex Positional Encodingcited by 1
  6. 2025Mitigating Coordinate Prediction Bias from Positional Encoding Failurescited by 1
  7. 2025OMEGA: Optimized Multimodal Position Encoding Index Derivation with Global Adaptive Scaling for Vision-Language Modelscited by 1
  8. 2026Towards Understanding What State Space Models Learn About Codecited by 1
  9. 2026Training-free Spatially Grounded Geometric Shape Encoding (Technical Report)cited by 1
  10. 2026RelFlexformer: Efficient Attention 3D-Transformers for Integrable Relative Positional Encodingscited by 1
  11. 2026V2VCrafter: Consistent Street-View Image Generation Across Vehiclescited by 1
  12. 2026How Data Shapes RoPE Frequency Usage: From Positional Scale Matching to Length Generalizationcited by 1
  13. 2025Sensitivity Meets Sparsity: The Impact of Extremely Sparse Parameter Patterns on Theory-of-Mind of Large Language Modelscited by 0
  14. 2025Beyond flattening: a geometrically principled positional encoding for vision transformers with Weierstrass elliptic functionscited by 0
  15. 2025Causality-Induced Positional Encoding for Transformer-Based Representation Learning of Non-Sequential Featurescited by 0
  16. 2025Towards Efficient Post-Training via Fourier-Driven Adapter Architecturescited by 0
  17. 2026Towards Infinite Length Extrapolation: A Unified Approachcited by 0
  18. 2026UCAN: Unified Convolutional Attention Network for Expansive Receptive Fields in Lightweight Super-Resolutioncited by 0
  19. 2026Wisteria: A Unified Multi-Scale Feature Learning Framework for DNA Language Modelcited by 0
  20. 2026Weierstrass Positional Encoding for Vision Transformerscited by 0
  21. 2026Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEscited by 0
  22. 2026Semidirect Fourier Delta Attention: Phase-Controlled Delta Memory with Constructive Chunk-WY Kernelscited by 0
  23. 2026Anti-Periodic Positional Encoding: M"obius Boundary Conditions Make In-Context Retrieval Reliablecited by 0
  24. 2026Higher-Dimensional Rotary Position Embeddingcited by 0
  25. 2026RoPE is Dead, Long Live RoPE: Towards Scalable Data-aware Positional Encodingscited by 0