mlsanityData to 7 Oct 2026

GyRot: Leveraging Hidden Synergy between Rotation and Fine-grained Group Quantization for Low-bit LLM Inference

arXiv 2607.27694 ↗First posted 30 Jul 2026cs.AR cs.LGcited by 0

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

Low-bit quantization is essential for efficient LLM inference, and both rotation and fine-grained group quantization have shown individual promise. However, their combination often leads to accuracy degradation or hardware overhead due to a mismatch between the global nature of rotation and the localized behavior of group scaling. We propose GyRot, a quantization framework and hardware accelerator that bridges this gap through algorithm-hardware co-design. GyRot introduces Coarse Rotation, Fine Grouping (CoRFiG) and Harmonic-Aligned Permutation (HAP) to enable cooperative integration of rotation and group quantization, enhancing quantizability while relaxing scaling factor precision. To further reduce hardware cost, we reformulate asymmetric quantization and introduce a zero-point rounding strategy that enables fully integer dequantization. Implemented on an INT4-based tensor PE architecture, GyRot achieves state-of-the-art 4-bit accuracy across LLaMA-family models, while delivering up to 3.4x speedup and 3.6x energy efficiency over baseline LLM accelerators. These results validate GyRot's practical effectiveness for scalable and energy-efficient LLM deployment.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 46 of its 58 references that are in the map. First posted on arXiv 30 Jul 2026.

Builds on

46 of 58 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024SpinQuant: LLM quantization with learned rotations26 May
  2. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving7 May
  3. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs30 Mar
  4. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Serving29 Oct

Other references · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2016Pointer Sentinel Mixture Modelscited by 2,992
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
30 more
  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  2. 2020GLU Variants Improve Transformercited by 1,724
  3. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  4. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  5. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  6. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  7. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
  8. 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 194
  9. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 165
  10. 2023Microscaling Data Formats for Deep Learningcited by 132
  11. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 116
  12. 2023RPTQ: Reorder-based Post-training Quantization for Large Language Modelscited by 112
  13. 2020GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inferencecited by 103
  14. 2025OstQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fittingcited by 94
  15. 2023OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantizationcited by 83
  16. 2023ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensationcited by 78
  17. 2021VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inferencecited by 59
  18. 2023With Shared Microexponents, A Little Shifting Goes a Long Waycited by 47
  19. 2022ANT: Exploiting Adaptive Numerical Data Type for Low-bit Deep Neural Network Quantizationcited by 42
  20. 2018Training DNNs with Hybrid Block Floating Pointcited by 39
  21. 2025FP4 All the Way: Fully Quantized Training of LLMscited by 37
  22. 2024ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residualscited by 32
  23. 2024Tender: Accelerating Large Language Models via Tensor Decomposition and Runtime Requantizationcited by 22
  24. 2024MicroScopiQ: Accelerating Foundational Models through Outlier-Aware Microscaling Quantizationcited by 22
  25. 2024AMXFP4: Taming Activation Outliers with Asymmetric Microscaling Floating-Point for 4-bit LLM Inferencecited by 17
  26. 2024DFRot: Achieving Outlier-Free and Massive Activation-Free for Rotated LLMs with Refined Rotationcited by 16
  27. 2024BitMoD: Bit-serial Mixture-of-Datatype LLM Accelerationcited by 12
  28. 2025M-ANT: Efficient Low-bit Group Quantization for LLMs via Mathematically Adaptive Numerical Typecited by 12
  29. 2023AFPQ: Asymmetric Floating Point Quantization for LLMscited by 7
  30. 2024Panacea: Novel DNN Accelerator using Accuracy-Preserving Asymmetric Quantization and Energy-Saving Bit-Slice Sparsitycited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026PrismQuant: Optimal Null-Space Rotations for Grouped Quantizers
  2. 2024MixPE: Quantization and Hardware Co-design for Efficient LLM Inference
  3. 2024Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Format
  4. 2026MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference
  5. 2026SPARQLe: Sub-Precision Activation Representation for Quantized LLM Inference ↗
  6. 2026GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling
  7. 2025LiquidGEMM: Hardware-Efficient W4A8 GEMM Kernel for High-Performance LLM Serving
  8. 2026Qift: Shift-Friendly No-Zero W2 Post-Training Quantization for Rotated W2A4/KV4 LLM Inference