# RoLA: Rotary-Positioned Low-Rank Linear Attention for Efficient Diffusion Transformers arXiv: https://arxiv.org/abs/2609.06712 · HTML: https://mlsanity.com/paper/2609.06712/ ## Key facts - Cited by 1 papers in the map (data through 14 Oct 2026). - Placed in Video generation, one of 14 directions in Diffusion models. - Builds on 29 of its 29 references that are in the map. - First posted on arXiv 6 Sep 2026. ## Placed in - Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/) ## Builds on (most cited first, 29 of 29) - [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022) - [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022) - [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021) - [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015) - [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025) - [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024) - [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020) - [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023) - [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024) - [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019) - [LTX-Video: Realtime Video Latent Diffusion](https://mlsanity.com/paper/2501.00103/) (2024) - [OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation](https://mlsanity.com/paper/2407.02371/) (2024) - [VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness](https://mlsanity.com/paper/2503.21755/) (2025) - [Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity](https://mlsanity.com/paper/2502.01776/) (2025) - [Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation](https://mlsanity.com/paper/2505.18875/) (2025) - [VSA: Faster Video Diffusion with Trainable Sparse Attention](https://mlsanity.com/paper/2505.13389/) (2025) - [SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformer](https://mlsanity.com/paper/2509.24695/) (2025) - [Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generation](https://mlsanity.com/paper/2506.19852/) (2025) - [SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attention](https://mlsanity.com/paper/2509.24006/) (2025) - [VMoBA: Mixture-of-Block Attention for Video Diffusion Models](https://mlsanity.com/paper/2506.23858/) (2025) - [Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformers](https://mlsanity.com/paper/2506.03065/) (2025) - [Efficient-vDiT: Efficient Video Diffusion Transformers With Attention Tile](https://mlsanity.com/paper/2502.06155/) (2025) - [SLA2: Sparse-Linear Attention with Learnable Routing and QAT](https://mlsanity.com/paper/2602.12675/) (2026) - [Linearized Relative Positional Encoding](https://mlsanity.com/paper/2307.09270/) (2023) - [SVG-EAR: Parameter-Free Linear Compensation for Sparse Video Generation via Error-aware Routing](https://mlsanity.com/paper/2603.08982/) (2026) - [Mixture of Distributions Matters: Dynamic Sparse Attention for Efficient Video Diffusion Transformers](https://mlsanity.com/paper/2601.11641/) (2026) - [SALAD: Achieve High-Sparsity Attention via Efficient Linear Attention Tuning for Video Diffusion Transformer](https://mlsanity.com/paper/2601.16515/) (2026) - [RoPeSLR: 3D RoPE-driven Sparse-LowRank Attention for Efficient Diffusion Transformers](https://mlsanity.com/paper/2605.20659/) (2026) - [SANA-Video 2.0: Hybrid Linear Attention with Attention Residuals for Efficient Video Generation](https://mlsanity.com/paper/2607.21553/) (2026) ## Built on by (most cited first, 1 of 1) - [SparkDiffusion: Mitigating the High-Sparsity Trap --- A Unified Framework for up to 265× Single-GPU Acceleration of Visual Generation](https://mlsanity.com/paper/2609.23153/) (2026) ## Abstract (arXiv) Diffusion Transformers (DiTs) achieve strong video generation quality, but their dense spatiotemporal self-attention scales quadratically with sequence length and quickly becomes the dominant inference bottleneck. Sparse low-rank hybrids alleviate this cost by combining a local sparse branch with a global compressed branch. In video DiTs equipped with 3D Rotary Position Embeddings (RoPE), the global branch faces a structural compatibility issue: when RoPE is applied before a nonlinear feature map, the rotation and nonlinearity generally do not commute, making it difficult to keep a query-independent linear summary while preserving relative rotary geometry. Existing work often sidesteps this issue by replacing genuine cross-token global aggregation with coordinate-conditioned surrogates or learnable absolute positional modules. These compromises can be effective, but they approximate relative decay from absolute coordinates and introduce extra positional parameters. We propose RoLA, a rotary-positioned low-rank linear-attention branch that keeps genuine cross-token aggregation while remaining compatible with a reusable linear summary. The design applies RoPE outside the nonlinear low-rank feature map and reuses a truncated subset of the pre-trained rotary schedule matched to the low-rank bottleneck. This yields a linear-time low-rank global branch with relative positional behavior by design and no additional positional parameters; the full sparse--low-rank module still includes the fixed-sparsity sparse branch. Experiments on open-source video DiTs show that the resulting method remains competitive in generation quality at 90% sparsity while achieving 2.63× end-to-end inference speedup on Wan2.1-14B (720p, 81 frames, measured on an NVIDIA H100 GPU).