mlsanityData to 7 Oct 2026

Learning the RoPEs: Better 2D and 3D Position Encodings with STRING

arXiv 2502.02562 ↗First posted 4 Feb 2025cs.LG cs.AI cs.CVcited by 19

Not on the map. Why?

Abstract (arXiv)

We introduce STRING: Separable Translationally Invariant Position Encodings. STRING extends Rotary Position Encodings, a recently proposed and widely used algorithm in large language models, via a unifying theoretical framework. Importantly, STRING still provides exact translation invariance, including token coordinates of arbitrary dimensionality, whilst maintaining a low computational footprint. These properties are especially important in robotics, where efficient 3D token representation is key. We integrate STRING into Vision Transformers with RGB(-D) inputs (color plus optional depth), showing substantial gains, e.g. in open-vocabulary object detection and for robotics controllers. We complement our experiments with a rigorous mathematical analysis, proving the universality of our methods.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 36 of its 58 references that are in the map. First posted on arXiv 4 Feb 2025.

Builds on

36 of 58 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2024The Llama 3 Herd of Modelscited by 14,192
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  10. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  11. 2020Rethinking Attention with Performerscited by 1,584
  12. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
24 more
  1. 2018Self-Attention with Relative Position Representationscited by 1,112
  2. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  3. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  4. 2024PaliGemma: A versatile 3B VLM for transfercited by 679
  5. 2022Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Itemscited by 601
  6. 2021ABO: Dataset and Benchmarks for Real-World 3D Object Understandingcited by 360
  7. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  8. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  9. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  10. 2024Rotary Position Embedding for Vision Transformercited by 226
  11. 2024ALOHA Unleashed: A Simple Recipe for Robot Dexteritycited by 128
  12. 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 111
  13. 2015Benchmarking in Manipulation Research: The YCB Object and Model Set and Benchmarking Protocolscited by 98
  14. 2022KERPLE: Kernelized Relative Positional Embedding for Length Extrapolationcited by 84
  15. 2020Learning to Encode Position for Transformer with Continuous Dynamical Modelcited by 61
  16. 2021Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encodingcited by 50
  17. 2022Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysiscited by 39
  18. 2021From block-Toeplitz matrices to differential equations on graphs: towards a general theory for scalable masked Transformerscited by 38
  19. 2021A Simple and Effective Positional Encoding for Transformerscited by 37
  20. 2021SHAPE: Shifted Absolute Position Embedding for Transformerscited by 37
  21. 2019Equivariant Transformer Networkscited by 35
  22. 2022Early or Late Fusion Matters: Efficient RGB-D Fusion in Vision Transformers for 3D Object Recognitioncited by 11
  23. 2023Revisiting Energy Based Models as Policies: Ranking Noise Contrastive Estimation and Interpolating Energy Modelscited by 9
  24. 2024Linear Transformer Topological Masking with Graph Random Featurescited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 11.

19 papers.

Sort:

  1. 2025Cameras as Relative Positional Encodingcited by 94
  2. 2026RayRoPE: Projective Ray Positional Encoding for Multi-view Attentioncited by 10
  3. 2025Group Representational Position Encodingcited by 5
  4. 2025Rethinking RoPE: A Mathematical Blueprint for N-dimensional Positional Embeddingcited by 3
  5. 2025A Circular Argument : Does RoPE need to be Equivariant for Vision?cited by 2
  6. 2026Adaptive 3D-RoPE: Physics-Aligned Rotary Positional Encoding for Wireless Foundation Modelscited by 2
  7. 2026nD-RoPE: A Generalized RoPE for n-Dimensional Position Embeddingcited by 2
  8. 2025Rotary Position Encodings for Graphscited by 1
  9. 2025Clifford Algebraic Rotor Embeddings : Maybe embeddings should start to CAREcited by 1
  10. 2026CLAMP: Contrastive Learning for 3D Multi-View Action-Conditioned Robotic Manipulation Pretrainingcited by 1
  11. 2026RelFlexformer: Efficient Attention 3D-Transformers for Integrable Relative Positional Encodingscited by 1
  12. 2025Linear Memory SE(2) Invariant Attentioncited by 0
7 more
  1. 2025Head-wise Adaptive Rotary Positional Encoding for Fine-Grained Image Generationcited by 0
  2. 2026LSP-DETR: Efficient and Scalable Nuclei Segmentation in Whole-Slide Imagescited by 0
  3. 2026Parabolic Position Encoding: Vision-Centric, Principled, Extrapolatable, Generalcited by 0
  4. 2026Do Vision Transformers Need All-to-All Attention? Global Communication Through Elastic Learned Corescited by 0
  5. 2026ClockRoPE: Random Fourier Rotations for Temporal Routine Modelingcited by 0
  6. 2026DINOcular: Self-Supervised Visuospatial Representationscited by 0
  7. 2026MinkowskiPE: Minkowski Positional Encoding for Spatiotemporal Perceptioncited by 0