# Cross-Axis Transformer with 3D Rotary Positional Embeddings

arXiv: https://arxiv.org/abs/2311.07184 · HTML: https://mlsanity.com/paper/2311.07184/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Nov 2023.

## Built on by (most cited first, 2 of 2)

- [A Survey of Retentive Network](https://mlsanity.com/paper/2506.06708/) (2025)
- [Mage: Cracking Elliptic Curve Cryptography with Cross-Axis Transformers](https://mlsanity.com/paper/2512.12483/) (2025)

## Abstract (arXiv)

Despite lagging behind their modal cousins in many respects, Vision Transformers have provided an interesting opportunity to bridge the gap between sequence modeling and image modeling. Up until now however, vision transformers have largely been held back, due to both computational inefficiency, and lack of proper handling of spatial dimensions. In this paper, we introduce the Cross-Axis Transformer. CAT is a model inspired by both Axial Transformers, and Microsoft's recent Retentive Network, that drastically reduces the required number of floating point operations required to process an image, while simultaneously converging faster and more accurately than the Vision Transformers it replaces.