mlsanityData to 14 Oct 2026

Rotary Position Embedding for Vision Transformer

arXiv 2403.13298 ↗First posted 20 Mar 2024cs.CV cs.LGcited by 236

Not on the map. Why?

Cited by 34 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Rotary Position Embedding (RoPE) performs remarkably on language models, especially for length extrapolation of Transformers. However, the impacts of RoPE on computer vision domains have been underexplored, even though RoPE appears capable of enhancing Vision Transformer (ViT) performance in a way similar to the language domain. This study provides a comprehensive analysis of RoPE when applied to ViTs, utilizing practical implementations of RoPE for 2D vision data. The analysis reveals that RoPE demonstrates impressive extrapolation performance, i.e., maintaining precision while increasing image resolution at inference. It eventually leads to performance improvement for ImageNet-1k, COCO detection, and ADE-20k segmentation. We believe this study provides thorough guidelines to apply RoPE into ViT, promising improved backbone performance with minimal extra computational overhead. Our code and pre-trained models are available at https://github.com/naver-ai/rope-vit

Key facts. Cited by 236 papers in the map (data through 14 Oct 2026). Builds on 34 of its 41 references that are in the map. First posted on arXiv 20 Mar 2024.

Builds on

34 of 41 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  10. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  11. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  12. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
22 more
  1. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  2. 2023Mistral 7Bcited by 3,164
  3. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  4. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  5. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,319
  6. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  7. 2018Self-Attention with Relative Position Representationscited by 1,120
  8. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  9. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 569
  10. 2022DeiT III: Revenge of the ViTcited by 419
  11. 2021Conditional Positional Encodings for Vision Transformerscited by 406
  12. 2023EVA-02: A Visual Representation for Neon Genesiscited by 381
  13. 2021ResNet strikes back: An improved training procedure in timmcited by 368
  14. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  15. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 303
  16. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  17. 2021Rethinking and Improving Relative Position Encoding for Vision Transformercited by 207
  18. 2022FlexiViT: One Model for All Patch Sizescited by 152
  19. 2023What Do Self-Supervised Vision Transformers Learn?cited by 93
  20. 2024FiT: Flexible Vision Transformer for Diffusion Modelcited by 85
  21. 2021CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddingscited by 47
  22. 2022ResFormer: Scaling ViTs with Multi-Resolution Trainingcited by 34

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 15; 2025: 82; 2026: 139.

236 papers.

Sort:

By area:

  1. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  2. 2025Qwen-Image Technical Reportcited by 1,044
  3. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
  4. 2024Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesiscited by 304
  5. 2026Gemma 4 Technical Reportcited by 259
  6. 2024Multimodal Whole Slide Foundation Model for Pathologycited by 162
  7. 2025Cameras as Relative Positional Encodingcited by 105
  8. 2025Point3R: Streaming 3D Reconstruction with Explicit Spatial Pointer Memorycited by 87
  9. 2024SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizercited by 56
  10. 2024Base of RoPE Bounds Context Lengthcited by 52
  11. 2026Qwen-RobotManip Technical Report: Alignment Unlocks Scale for Robotic Manipulation Foundation Modelscited by 50
  12. 2024Towards Semantic Equivalence of Tokenization in Multimodal LLMcited by 45
38 more
  1. 2025Reflect-DiT: Inference-Time Scaling for Text-to-Image Diffusion Transformers via In-Context Reflectioncited by 43
  2. 2024Switti: Designing Scale-Wise Transformers for Text-to-Image Synthesiscited by 33
  3. 2025OctGPT: Octree-based Multiscale Autoregressive Models for 3D Shape Generationcited by 30
  4. 2025Why Do MLLMs Struggle with Spatial Understanding? A Systematic Analysis from Data to Architecturecited by 30
  5. 2025Visual Autoregressive Modeling for Instruction-Guided Image Editingcited by 22
  6. 2025Generalized and Efficient 2D Gaussian Splatting for Arbitrary-scale Super-Resolutioncited by 21
  7. 2025MDNS: Masked Diffusion Neural Sampler via Stochastic Optimal Controlcited by 21
  8. 2026Utonia: Toward One Encoder for All Point Cloudscited by 21
  9. 2025Learning the RoPEs: Better 2D and 3D Position Encodings with STRINGcited by 19
  10. 2025Lost in Latent Space: An Empirical Study of Latent Diffusion Models for Physics Emulationcited by 19
  11. 2024LieRE: Lie Rotational Positional Encodingscited by 18
  12. 2025ComRoPE: Scalable and Robust Rotary Position Embedding Parameterized by Trainable Commuting Angle Matricescited by 18
  13. 2025X-Streamer: Unified Human World Modeling with Audiovisual Interactioncited by 18
  14. 2025Proximal Diffusion Neural Samplercited by 16
  15. 2026Addressable Memory for Video World Modelscited by 15
  16. 2025RoPECraft: Training-Free Motion Transfer with Trajectory-Guided RoPE Optimization on Diffusion Transformerscited by 14
  17. 2025DyPE: Dynamic Position Extrapolation for Ultra High Resolution Diffusioncited by 14
  18. 2026RayRoPE: Projective Ray Positional Encoding for Multi-view Attentioncited by 14
  19. 2026Qwen-RobotWorld Technical Report: Unifying Embodied World Modeling through Language-Conditioned Video Generationcited by 14
  20. 2025EchoMotion: Unified Human Video and Motion Generation via Dual-Modality Diffusion Transformercited by 13
  21. 2026ReRoPE: Repurposing RoPE for Relative Camera Controlcited by 13
  22. 2024FuXi-2.0: Advancing machine learning weather forecasting model for practical applicationscited by 11
  23. 2025ReDirector: Creating Any-Length Video Retakes with Rotary Camera Encodingcited by 11
  24. 2024LookHere: Vision Transformers with Directed Attention Generalize and Extrapolatecited by 10
  25. 2025MMTABREAL: Real-World Benchmark for Multimodal Table Understandingcited by 9
  26. 2026HyperDiT: Hyper-Connected Transformers for High-Fidelity Pixel-Space Diffusioncited by 9
  27. 2025RiemannFormer: A Framework for Attention in Curved Spacescited by 8
  28. 2025PanoSplatt3R: Leveraging Perspective Pretraining for Generalized Unposed Wide-Baseline Panorama Reconstructioncited by 8
  29. 2025AERIS: Argonne Earth Systems Model for Reliable and Skillful Predictionscited by 8
  30. 2025ARROW: An Adaptive Rollout and Routing Method for Global Weather Forecastingcited by 8
  31. 2025InfoTok: Adaptive Discrete Video Tokenizer via Information-Theoretic Compressioncited by 8
  32. 2026UniDAC: Universal Metric Depth Estimation for Any Cameracited by 8
  33. 2026Spiral RoPE: Rotate Your Rotary Positional Embeddings in the 2D Planecited by 7
  34. 2025LaDCast: A Latent Diffusion Model for Medium-Range Ensemble Weather Forecastingcited by 6
  35. 2025Lizard: An Efficient Linearization Framework for Large Language Modelscited by 6
  36. 2025LVT: Large-Scale Scene Reconstruction via Local View Transformerscited by 6
  37. 2025NAF: Zero-Shot Feature Upsampling via Neighborhood Attention Filteringcited by 6
  38. 2026URoPE: Universal Relative Position Embedding across Geometric Spacescited by 6

50 most cited of 236 All 236 (JSON)