mlsanityData to 14 Oct 2026

WUSH: Near-Optimal Adaptive Transforms for LLM Quantization

arXiv 2512.00956 ↗First posted 30 Nov 2025cs.LG cs.CLcited by 15

Not on the map. Why?

Abstract (arXiv)

Quantizing LLM weights and activations is a standard approach for efficient deployment, but a few extreme outliers can stretch the dynamic range and amplify low-bit quantization errors. Prior transform-based mitigations (e.g., Hadamard rotations) are fixed and data-agnostic, and their optimality for quantization has remained unclear. We derive closed-form optimal linear blockwise transforms for joint weight-activation quantization under standard RTN AbsMax-scaled block quantizers, covering both integer and floating-point formats. The resulting construction, WUSH, combines a Hadamard backbone with a data-dependent second-moment component to form a non-orthogonal transform that is provably near-optimal for FP and INT quantizers under mild assumptions while admitting an efficient fused GPU implementation. Empirically, WUSH improves W4A4 accuracy over the strongest Hadamard-based baselines (e.g., on Llama-3.1-8B-Instruct in MXFP4, it gains +2.8 average points with RTN and +0.7 with GPTQ) while delivering up to 5.8× per-layer throughput over BF16 via FP4 MatMul. Source code is available at https://github.com/IST-DASLab/WUSH.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 18 of its 26 references that are in the map. First posted on arXiv 30 Nov 2025.

Builds on

18 of 26 references.

References · 18

  1. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  2. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  3. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  4. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  5. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  6. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  7. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  8. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  9. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 322
  10. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 131
  11. 2024QTIP: Quantization with Trellises and Incoherence Processingcited by 91
  12. 2025Do Large Language Model Benchmarks Test Reliability?cited by 54
6 more
  1. 2021Differentiable Model Compression via Pseudo Quantization Noisecited by 45
  2. 2023QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Modelscited by 39
  3. 2024"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantizationcited by 34
  4. 2025Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantizationcited by 32
  5. 2025The Geometry of LLM Quantization: GPTQ as Babai's Nearest Plane Algorithmcited by 20
  6. 2025Block Rotation is All You Need for MXFP4 Quantizationcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 15.

15 papers.

Sort:

By area:

  1. 2026High-Rate Quantized Matrix Multiplication Icited by 7
  2. 2026Dissecting Quantization Error: A Concentration-Alignment Perspectivecited by 5
  3. 2026HARP: Hadamard-Preconditioned Adaptive Rotation Processor for Extreme LLM Quantizationcited by 5
  4. 2026Adaptive Block-Scaled Data Typescited by 4
  5. 2026Price of metric universality in vector quantization is at most 0.11 bitcited by 1
  6. 2026A Nuclear-Norm Lower Bound for Dithered Scalar Quantization of Matrix Productscited by 1
  7. 2026When Local Variance Optimality Is Not Enough: RoPE-Aligned Q/K Rotations for Dynamic 4-Bit Quantisationcited by 0
  8. 2026KBBQ: A Predictive Noise Law and the Limits of Spectrum Flattening in FP4 Quantizationcited by 0
  9. 2026Product-Aware Deterministic Rounding for Quantized Matrix Multiplicationcited by 0
  10. 2026QuantForge: Discovering Residual Decompositions for MXFP4 Post-Training Quantizationcited by 0
  11. 2026WUSH-KV: KV Cache Quantization with Data-Adaptive Transformscited by 0
  12. 2026The Devil Is in the Reconstruction Loss Scale: Rethinking Optimization in LLM Quantizationcited by 0
3 more
  1. 2026OSFP4: Joint Optimization of Diagonal Smoothing and Block Scales for NVFP4 Quantizationcited by 0
  2. 2026WUSPruner: Second-Moment-Optimal Transforms for Structured Pruningcited by 0
  3. 2026SpectrumQuant: Spectrum Alignment for LLM Quantizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ButterflyQuant: Ultra-low-bit LLM Quantization through Learnable Orthogonal Butterfly Transforms
  2. 2024DFRot: Achieving Outlier-Free and Massive Activation-Free for Rotated LLMs with Refined Rotation
  3. 2026Transforms for LLM Quantization: The Great Inversion and Format Co-Design
  4. 2023LLM-FP4: 4-Bit Floating-Point Quantized Transformers
  5. 2025SpecQuant: Spectral Decomposition and Adaptive Truncation for Ultra-Low-Bit LLMs Quantization
  6. 2026Breaking the Blocks: Continuous Low-Rank Decomposed Scaling for Unified LLM Quantization and Adaptation
  7. 2026QAM-W: Joint 2D Codebook Quantization for LLM Weights via Hadamard Rotation and Activation-Aware Scaling
  8. 2024Taming Sensitive Weights : Noise Perturbation Fine-tuning for Robust LLM Quantization