mlsanityData to 14 Oct 2026

Influence-Inspired Spectral Rotations for Extreme Low-Bit LLM Quantization

arXiv 2605.25203 ↗First posted 24 May 2026cs.LG cs.AI cs.LOcited by 1

Not on the map. Why?

Abstract (arXiv)

We apply the influence-adaptive Walsh geometry of a companion theory paper (arXiv:2605.01637) to extreme low-bit weight-only LLM quantization. The recipe is one math-invariant transformation: WHT-rotate each linear layer's weight matrix and rescale its columns by per-coordinate Walsh-basis activation energy before handing off to a reconstruction-error quantizer (Intel auto-round). This biases per-group integer rounding toward high-spectral-energy channels. On four pretrained decoder-only models from 135M to 1.5B parameters, BBT-spectral reduces wikitext-2 perplexity by 15-58% relative to vanilla auto-round at W2A16; we also report a TinyLlama-1.1B auxiliary data point. Three extensions transfer the recipe to families it failed on: a per-head PCA matrix-Gamma replacement of q_norm/k_norm for Qwen3 attention (PPL 136.76 -> 88.99 on Qwen3-0.6B); an SO(2) per-pair rotation that commutes with RoPE (PPL 36.93 -> 21.84 on Qwen2.5-1.5B); and an MoE-aware input-side absorption fix identified by architectural fuzzing of Laguna-style fused-expert layouts. A W2-vs-W4 ablation gives a deliberate negative control: the redistribution payoff falls within the +/-0.5 PPL noise floor at W4, consistent with the Schur-convexity intuition that the cost of unconcentrated influence vanishes as the noise budget shrinks. All quantized weights export to OpenVINO IR and run on Intel NPU + Arc dGPU + CPU with PPL invariant to device within +/-0.1. We do not claim a formal Boolean-to-real-valued transfer of the theory paper's majorization argument: the WHT activation energy used here is not the Boolean influence of the theory paper, the link is intuitive, and the contribution is engineering value rather than a transferred theorem. Head-to-head benchmarks against SpinQuant, QuaRot, QuIP-sharp, AQLM, OmniQuant, and ButterflyQuant at matched calibration are the main future-work item.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 16 of its 19 references that are in the map. First posted on arXiv 24 May 2026.

Builds on

16 of 19 references.

References · 16

  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  2. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  3. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  4. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  5. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  6. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  7. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  8. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  9. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  10. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  11. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
  12. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
4 more
  1. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 189
  2. 2023Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMscited by 50
  3. 2025ButterflyQuant: Ultra-low-bit LLM Quantization through Learnable Orthogonal Butterfly Transformscited by 10
  4. 2026The Banach-Butterfly Invariant: Influence-Adaptive Walsh Geometry for Ternary Polynomial Threshold Functionscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026When Local Variance Optimality Is Not Enough: RoPE-Aligned Q/K Rotations for Dynamic 4-Bit Quantisationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SpecQuant: Spectral Decomposition and Adaptive Truncation for Ultra-Low-Bit LLMs Quantization
  2. 2026LittleBit-2: Maximizing the Spectral Energy Gain in Sub-1-Bit LLMs via Latent Geometry Alignment
  3. 2026Qift: Shift-Friendly No-Zero W2 Post-Training Quantization for Rotated W2A4/KV4 LLM Inference
  4. 2026QAM-W: Joint 2D Codebook Quantization for LLM Weights via Hadamard Rotation and Activation-Aware Scaling
  5. 2026Breaking the Blocks: Continuous Low-Rank Decomposed Scaling for Unified LLM Quantization and Adaptation
  6. 2025WUSH: Near-Optimal Adaptive Transforms for LLM Quantization
  7. 2026HARP: Hadamard-Preconditioned Adaptive Rotation Processor for Extreme LLM Quantization
  8. 2026Transforms for LLM Quantization: The Great Inversion and Format Co-Design