RotateAttention: RoPE-Aware Rotation and Range Rectification for INT4 Quantized Attention in Video Generation

In DiT-based video generation models equipped with 3D Rotary Position Embeddings (3D RoPE), the attention mechanism remains a primary computational bottleneck due to its quadratic complexity with respect to sequence length. While quantized FlashAttention offers a promising path toward hardware acceleration, existing low-bit quantization methods overlook two critical challenges in this setting: 1) applying online rotation matrices -- a widely used technique for mitigating outliers in Queries (Q) and Keys (K) -- is difficult to reconcile with RoPE; and 2) the non-negative attention matrix P = exp(QK - max(QK)) makes symmetric quantization waste half of the 4-bit dynamic range. In this work, we observe that the outlier distributions of Q and K are strongly affected by the dimensional partitioning of 3D RoPE. Based on this finding, we propose RotateAttention, an efficient mixed-precision INT4 FlashAttention framework tailored for DiT-based video generation models with 3D RoPE, using selective FP16 fallback for accuracy-sensitive attention blocks and denoising steps. RotateAttention introduces two core techniques: 1) RoPE-aware Rotation, which employs either mergeable rotation matrices that can be fused into RoPE or negligible-overhead matrices to mitigate RoPE-induced outliers in Q and K; and 2) Range-optimized P Quantization, which uses fixed scales and zero-points to fully exploit the INT4 numerical range with minimal computational overhead. Experiments show that RotateAttention preserves video generation quality nearly identical to full-precision baselines while achieving up to 1.68× end-to-end speedup and 2.2× kernel-level acceleration.