mlsanityData to 7 Oct 2026

GaugeQuant: Online Learning of Quantization-Optimal Bases from LLM Symmetries

arXiv 2607.20757 ↗First posted 22 Jul 2026cs.LG cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Transformers are known to have internal continuous symmetries that leave outputs invariant, while modifying quantization. GaugeQuant leverages this in-training by introducing a LogSumExp term to the loss that breaks the symmetries, thus selecting a basis that minimizes activation outliers. A stop-gradient operator ensures that only rotation matrices are updated, yielding the language modeling objective completely unaltered. Our requires no specific calibration data, no quantization simulation, and adds negligible training overhead. With the LLaMA-2 7B model under W4A4 quantization with group size 128, perplexity drops from 8.22 to 6.73, competing with post-training methods that require frozen models and calibration datasets. Under W4A16, perplexity drops from 11.16 to 5.45. Code is available at https://github.com/MPedraBento/gauge-quant.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 22 of its 26 references that are in the map. First posted on arXiv 22 Jul 2026.

Builds on

22 of 26 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  8. 2020GLU Variants Improve Transformercited by 1,724
  9. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  10. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  11. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  12. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
10 more
  1. 2024SpinQuant: LLM quantization with learned rotationscited by 446
  2. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378
  3. 2020Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transformcited by 79
  4. 2025ParoQuant: Pairwise Rotation Quantization for Efficient Reasoning LLM Inferencecited by 16
  5. 2025HALO: Hadamard-Assisted Lower-Precision Optimization for LLMscited by 14
  6. 2024RoLoRA: Fine-tuning Rotated Outlier-free LLMs for Effective Weight-Activation Quantizationcited by 13
  7. 2025Outlier-Safe Pre-Training for Robust 4-Bit Quantization of Large Language Modelscited by 13
  8. 2024Unification of Symmetries Inside Neural Networks: Transformer, Feedforward and Neural ODEcited by 6
  9. 2025RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Modelscited by 5
  10. 2026ReSpinQuant: Efficient Layer-Wise LLM Quantization via Subspace Residual Rotation Approximationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Softmax Reparameterization for Output-Head Quantizationcited by 0