mlsanityData to 14 Oct 2026

Grouped Sequency-arranged Rotation: Optimizing Rotation Transformation for Quantization for Free

arXiv 2505.03810 ↗First posted 2 May 2025cs.LG cs.AI cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) face deployment challenges due to high computational costs, and while Post-Training Quantization (PTQ) offers a solution, existing rotation-based methods struggle at very low bit-widths like 2-bit. We introduce a novel, training-free approach to construct an improved rotation matrix, addressing the limitations of current methods. The key contributions include leveraging the Walsh-Hadamard transform with sequency ordering, which clusters similar frequency components to reduce quantization error compared to standard Hadamard matrices, significantly improving performance. Furthermore, we propose a Grouped Sequency-arranged Rotation (GSR) using block-diagonal matrices with smaller Walsh blocks, effectively isolating outlier impacts and achieving performance comparable to optimization-based methods without requiring any training. Our method demonstrates robust performance on reasoning tasks and Perplexity (PPL) score on WikiText-2. Our method also enhances results even when applied over existing learned rotation techniques.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 13 of its 24 references that are in the map. First posted on arXiv 2 May 2025.

Builds on

13 of 24 references.

References · 13

  1. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  2. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  3. 2016Pointer Sentinel Mixture Modelscited by 3,096
  4. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  5. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  6. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  7. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  8. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  9. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  10. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  11. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 167
  12. 2025OstQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fittingcited by 97
1 more
  1. 2024DFRot: Achieving Outlier-Free and Massive Activation-Free for Rotated LLMs with Refined Rotationcited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantizationcited by 32
  2. 2026Decomposing MXFP4 quantization error for LLM reinforcement learning: reducible bias, recoverable deadzone, and an irreducible floorcited by 1
  3. 2026When Local Variance Optimality Is Not Enough: RoPE-Aligned Q/K Rotations for Dynamic 4-Bit Quantisationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025BASE-Q: Bias and Asymmetric Scaling Enhanced Rotational Quantization for Large Language Models
  2. 2026ThinQuant: Scalable Rotation Learning for Weight and Activation Quantization of LLMs
  3. 2025Gradual Binary Search and Dimension Expansion : A general method for activation quantization in LLMs
  4. 2025Outlier Smoothing with Closed-Form Rotations for W4A4 Large Language Model Quantization
  5. 2026ReSpinQuant: Efficient Layer-Wise LLM Quantization via Subspace Residual Rotation Approximation
  6. 2026HARP: Hadamard-Preconditioned Adaptive Rotation Processor for Extreme LLM Quantization
  7. 2025RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotations
  8. 2026ConQuR: Corner Aligned Activation Quantization via Optimized Rotations for LLMs