mlsanityData to 7 Oct 2026

RoFormer: Enhanced Transformer with Rotary Position Embedding

arXiv 2104.09864 ↗First posted 20 Apr 2021cs.CL cs.AI cs.LGcited by 4,581

Not on the map. Why?

Cited by 545 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Position encoding recently has shown effective in the transformer architecture. It enables valuable supervision for dependency modeling between elements at different positions of the sequence. In this paper, we first investigate various methods to integrate positional information into the learning process of transformer-based language models. Then, we propose a novel method named Rotary Position Embedding(RoPE) to effectively leverage the positional information. Specifically, the proposed RoPE encodes the absolute position with a rotation matrix and meanwhile incorporates the explicit relative position dependency in self-attention formulation. Notably, RoPE enables valuable properties, including the flexibility of sequence length, decaying inter-token dependency with increasing relative distances, and the capability of equipping the linear self-attention with relative position encoding. Finally, we evaluate the enhanced transformer with rotary position embedding, also called RoFormer, on various long text classification benchmark datasets. Our experiments show that it consistently overcomes its alternatives. Furthermore, we provide a theoretical analysis to explain some experimental results. RoFormer is already integrated into Huggingface: urlhttps://huggingface.co/docs/transformers/modeldoc/roformer.

Key facts. Cited by 4,581 papers in the map (data through 7 Oct 2026). Builds on 36 of its 55 references that are in the map. First posted on arXiv 20 Apr 2021.

Builds on

36 of 55 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2018Neural Ordinary Differential Equationscited by 3,733
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  10. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  11. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  12. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
24 more
  1. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  2. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  3. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  4. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  5. 2020Rethinking Attention with Performerscited by 1,584
  6. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  7. 2018Self-Attention with Relative Position Representationscited by 1,112
  8. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  9. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  10. 2017Convolutional Sequence to Sequence Learningcited by 550
  11. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
  12. 2018Efficient Attention: Attention with Linear Complexitiescited by 366
  13. 2016A Decomposable Attention Model for Natural Language Inferencecited by 214
  14. 2018Music Transformercited by 207
  15. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  16. 2020How Much Position Information Do Convolutional Neural Networks Encode?cited by 171
  17. 2018Training Tips for the Transformer Modelcited by 85
  18. 2020Improve Transformer Models with Better Relative Position Embeddingscited by 73
  19. 2020Learning to Encode Position for Transformer with Continuous Dynamical Modelcited by 61
  20. 2019Encoding word order in complex embeddingscited by 53
  21. 2019NEZHA: Neural Contextualized Representation for Chinese Language Understandingcited by 39
  22. 2019CAIL2019-SCM: A Dataset of Similar Case Matching in Legal Domaincited by 30
  23. 2016Is Neural Machine Translation Ready for Deployment? A Case Study on 30 Translation Directionscited by 7
  24. 2021Translational Equivariance in Kernelizable Attentioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 19; 2022: 68; 2023: 259; 2024: 875; 2025: 1582; 2026: 1778.

4,581 papers.

Sort:

By area:

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  7. 2023Qwen Technical Reportcited by 3,273
  8. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
  9. 2023A Survey of Large Language Modelscited by 2,537
  10. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  11. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  12. 2023Code Llama: Open Foundation Models for Codecited by 2,105
38 more
  1. 2024Qwen2 Technical Reportcited by 2,076
  2. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  3. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  5. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  6. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  7. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  8. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  9. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  10. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  11. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  12. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  13. 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 960
  14. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 928
  15. 2023BloombergGPT: A Large Language Model for Financecited by 923
  16. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  17. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  18. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  19. 2025Large Language Diffusion Modelscited by 872
  20. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  21. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  22. 2024Simple and Effective Masked Diffusion Language Modelscited by 800
  23. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  24. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  25. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  26. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  27. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  28. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 712
  29. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  30. 2024Yi: Open Foundation Models by 01.AIcited by 706
  31. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  32. 2024A Survey on Large Language Models for Code Generationcited by 646
  33. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  34. 2023A Comprehensive Overview of Large Language Modelscited by 639
  35. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  36. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  37. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  38. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598

50 most cited of 4,581 All 4,581 (JSON)