mlsanityData to 7 Oct 2026

CRoPE: Efficient Parametrization of Rotary Positional Embedding

arXiv 2601.02728 ↗First posted 6 Jan 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Rotary positional embedding has become the state-of-the-art approach to encode position information in transformer-based models. While it is often succinctly expressed in complex linear algebra, we note that the actual implementation of Q/K/V-projections is not equivalent to a complex linear transformation. We argue that complex linear transformation is a more natural parametrization and saves near 50% parameters within the attention block. We show empirically that removing such redundancy has negligible impact on the model performance. Our modification achieves more efficient parameter usage, as well as a cleaner interpretation of the representation space.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 25 of its 36 references that are in the map. First posted on arXiv 6 Jan 2026.

Builds on

25 of 36 references.

References · 25

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  8. 2016Pointer Sentinel Mixture Modelscited by 2,992
  9. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  11. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  12. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
13 more
  1. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  2. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  3. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  4. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  5. 2022Confident Adaptive Language Modelingcited by 297
  6. 2017Deep Complex Networkscited by 215
  7. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
  8. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  9. 2020AutoML-Zero: Evolving Machine Learning Algorithms From Scratchcited by 137
  10. 2021Position Information in Transformers: An Overviewcited by 103
  11. 2019Encoding word order in complex embeddingscited by 53
  12. 2021A Simple and Effective Positional Encoding for Transformerscited by 37
  13. 2024ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residualscited by 32

Built on by

Not cited yet.