# RoPeSLR: 3D RoPE-driven Sparse-LowRank Attention for Efficient Diffusion Transformers

arXiv: https://arxiv.org/abs/2605.20659 · HTML: https://mlsanity.com/paper/2605.20659/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 25 of its 32 references that are in the map.
- First posted on arXiv 20 May 2026.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 25 of 25)

- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers](https://mlsanity.com/paper/2410.10629/) (2024)
- [OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation](https://mlsanity.com/paper/2407.02371/) (2024)
- [Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity](https://mlsanity.com/paper/2502.01776/) (2025)
- [SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration](https://mlsanity.com/paper/2410.02367/) (2024)
- [SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inference](https://mlsanity.com/paper/2502.18137/) (2025)
- [VSA: Faster Video Diffusion with Trainable Sparse Attention](https://mlsanity.com/paper/2505.13389/) (2025)
- [VDT: General-purpose Video Diffusion Transformers via Mask Modeling](https://mlsanity.com/paper/2305.13311/) (2023)
- [Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generation](https://mlsanity.com/paper/2506.19852/) (2025)
- [SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attention](https://mlsanity.com/paper/2509.24006/) (2025)
- [Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformers](https://mlsanity.com/paper/2506.03065/) (2025)
- [Breaking the Low-Rank Dilemma of Linear Attention](https://mlsanity.com/paper/2411.07635/) (2024)
- [LiteAttention: A Temporal Sparse Attention for Diffusion Transformers](https://mlsanity.com/paper/2511.11062/) (2025)
- [Re-ttention: Ultra Sparse Visual Generation via Attention Statistical Reshape](https://mlsanity.com/paper/2505.22918/) (2025)
- [Mixture of Distributions Matters: Dynamic Sparse Attention for Efficient Video Diffusion Transformers](https://mlsanity.com/paper/2601.11641/) (2026)
- [CR-Net: Scaling Parameter-Efficient Training with Cross-Layer Low-Rank Structure](https://mlsanity.com/paper/2509.18993/) (2025)
- [SALAD: Achieve High-Sparsity Attention via Efficient Linear Attention Tuning for Video Diffusion Transformer](https://mlsanity.com/paper/2601.16515/) (2026)

## Built on by (most cited first, 4 of 4)

- [RoLA: Rotary-Positioned Low-Rank Linear Attention for Efficient Diffusion Transformers](https://mlsanity.com/paper/2609.06712/) (2026)
- [SCOPE: Subspace Clustering with Online Per-Head Top-K Estimation for Sparse Video Attention](https://mlsanity.com/paper/2608.12780/) (2026)
- [SparkDiffusion: Mitigating the High-Sparsity Trap --- A Unified Framework for up to 265× Single-GPU Acceleration of Visual Generation](https://mlsanity.com/paper/2609.23153/) (2026)
- [Parameterized Stripe Attention for Efficient Video Generation](https://mlsanity.com/paper/2609.37001/) (2026)

## Abstract (arXiv)

Diffusion Transformers (DiTs) have revolutionized high-fidelity video generation, yet their O(L2) attention complexity poses a formidable bottleneck for long-sequence synthesis. While recent sparse-linear attention hybrids aim to mitigate this, their performance severely degrades at extreme sparsity due to the "RoPE Dilemma": standard linear attention fails to preserve the orthogonal relative-position structure of 3D Rotary Position Embeddings (RoPE), neutralizing vital distance awareness. To address this, we propose RoPeSLR, a 3D RoPE-guided Sparse-LowRank attention framework. We establish that under empirically validated assumptions, the DiT attention manifold admits a decoupling into a high-frequency semantic spike set (bounded by O(L3/2) sparsity) and an extreme low-rank (O(dh log L)) background continuum. Guided by this structural prior, RoPeSLR eschews standard linear attention for a head-wise low-rank parameterization equipped with a learnable 3D Absolute Positional Embedding (PE) injection, seamlessly synthesizing long-range relative distance decay. By guaranteeing sub-quadratic sparsity and sub-linear rank growth, RoPeSLR is exceptionally suited for scaling to ultra-long video inference. Extensive evaluations validate this scalable superiority: at 90% sparsity, RoPeSLR achieves up to 10× fewer FLOPs on Wan2.1-1.3B and delivers a 2.26× end-to-end inference speedup on the ultra-long 100K+ token sequences of HunyuanVideo-13B, all while maintaining near-lossless generation fidelity (less than 1.3% average VBench degradation).