# DFRot: Achieving Outlier-Free and Massive Activation-Free for Rotated LLMs with Refined Rotation

arXiv: https://arxiv.org/abs/2412.00648 · HTML: https://mlsanity.com/paper/2412.00648/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Dec 2024.

## Built on by (most cited first, 17 of 17)

- [QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Models](https://mlsanity.com/paper/2510.16292/) (2025)
- [CogDDN: A Cognitive Demand-Driven Navigation with Decision Optimization and Dual-Process Thinking](https://mlsanity.com/paper/2507.11334/) (2025)
- [Kelle: Co-design KV Caching and eDRAM for Efficient LLM Serving in Edge Computing](https://mlsanity.com/paper/2510.16040/) (2025)
- [SmoothRot: Combining Channel-Wise Scaling and Rotation for Quantization-Friendly LLMs](https://mlsanity.com/paper/2506.05413/) (2025)
- [Grouped Sequency-arranged Rotation: Optimizing Rotation Transformation for Quantization for Free](https://mlsanity.com/paper/2505.03810/) (2025)
- [WSVD: Weighted Low-Rank Approximation for Fast and Efficient Execution of Low-Precision Vision-Language Models](https://mlsanity.com/paper/2604.02570/) (2026)
- [ConQuR: Corner Aligned Activation Quantization via Optimized Rotations for LLMs](https://mlsanity.com/paper/2605.10793/) (2026)
- [OffQ: Taming Structured Outliers in LLM Quantization by Offsetting](https://mlsanity.com/paper/2606.07116/) (2026)
- [Gradual Binary Search and Dimension Expansion : A general method for activation quantization in LLMs](https://mlsanity.com/paper/2504.13989/) (2025)
- [ESCA: Enabling Seamless Codec Avatar Execution through Algorithm and Hardware Co-Optimization for Virtual Reality](https://mlsanity.com/paper/2510.24787/) (2025)
- [CodeQuant: Unified Clustering and Quantization for Enhanced Outlier Smoothing in Low-Precision Mixture-of-Experts](https://mlsanity.com/paper/2604.10496/) (2026)
- [LASER: Loss-Aware Singular-value Decomposition and Rank Allocation for Efficient Low-Precision Vision-Language Models](https://mlsanity.com/paper/2606.00573/) (2026)
- [GyRot: Leveraging Hidden Synergy between Rotation and Fine-grained Group Quantization for Low-bit LLM Inference](https://mlsanity.com/paper/2607.27694/) (2026)
- [When Local Variance Optimality Is Not Enough: RoPE-Aligned Q/K Rotations for Dynamic 4-Bit Quantisation](https://mlsanity.com/paper/2608.13365/) (2026)
- [HBQ: Hierarchical Scaling Block Quantization with Hardware-Efficiency-Aware Design for Accurate LLM Inference](https://mlsanity.com/paper/2609.00450/) (2026)
- [Understanding LLM Quantization through Activation-Guided Compensation and Orthogonal Residuals](https://mlsanity.com/paper/2609.21450/) (2026)
- [PrismQuant: Optimal Null-Space Rotations for Grouped Quantizers](https://mlsanity.com/paper/2609.32429/) (2026)

## Abstract (arXiv)

Rotating the activation and weight matrices to reduce the influence of outliers in large language models (LLMs) has recently attracted significant attention, particularly in the context of model quantization. Prior studies have shown that in low-precision quantization scenarios, such as 4-bit weights and 4-bit activations (W4A4), randomized Hadamard transforms can achieve significantly higher accuracy than randomized orthogonal transforms. Notably, the reason behind this phenomenon remains unknown. In this paper, we find that these transformations show substantial improvement in eliminating outliers for common tokens and achieve similar quantization error. The primary reason for the accuracy difference lies in the fact that randomized Hadamard transforms can slightly reduce the quantization error for tokens with massive activations while randomized orthogonal transforms increase the quantization error. Due to the extreme rarity of these tokens and their critical impact on model accuracy, we consider this a long-tail optimization problem, and therefore construct a simple yet effective method: a weighted loss function. Additionally, we propose an optimization strategy for the rotation matrix that involves alternating optimization of quantization parameters while employing orthogonal Procrustes transforms to refine the rotation matrix. This makes the distribution of the rotated activation values more conducive to quantization, especially for tokens with massive activations. Our method enhances the Rotated LLMs by achieving dual free, Outlier-Free and Massive Activation-Free, dubbed as DFRot. Extensive experiments demonstrate the effectiveness and efficiency of DFRot. By tuning the rotation matrix using just a single sample, DFRot achieves a perplexity improvement of 0.98 and 0.95 on W4A4KV4 and W4A4KV16, respectively, for LLaMA3-70B, a model known for its quantization challenges.