mlsanityData to 14 Oct 2026

OSC: Hardware Efficient W4A4 Quantization via Outlier Separation in Channel Dimension

arXiv 2604.12782 ↗First posted 14 Apr 2026cs.LG cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

While 4-bit quantization is essential for high-throughput deployment of Large Language Models, activation outliers often lead to significant accuracy degradation due to the restricted dynamic range of low-bit formats. In this paper, we systematically investigate the spatial distribution of outliers and demonstrate a token-persistent structural clustering effect, where high-magnitude outliers consistently occupy fixed channels across tokens. Building on this insight, we propose OSC, a hardware-efficient framework for outlier suppression. During inference, OSC executes a dual-path computation consisting of a low-precision 4-bit General Matrix Multiplication (GEMM) path and a high-precision 16-bit branch GEMM path. Specifically, OSC uses an offline group-wise strategy to identify the channels where outliers are located and then performs structured sub-tensor extraction to coalesce these scattered activation channels into a compact dense tensor online. This mechanism implements outlier protection through regularized and high-throughput GEMM operations, achieving a seamless fit with modern 4-bit micro-scaling hardware. Furthermore, for the inputs of W2 where outlier clustering is less pronounced, we integrate a fallback strategy to FP8. Evaluation on Qwen3-8B and Qwen3-30B restricts the average accuracy drop to 2.19 and 1.12 points, respectively. Notably, OSC is highly hardware-friendly, achieving a peak speedup of 1.78x over the W8A8 GEMM baseline on a modern AI accelerator.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 19 of its 23 references that are in the map. First posted on arXiv 14 Apr 2026.

Builds on

19 of 23 references.

References · 19

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  7. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  8. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
  9. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  10. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  11. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  12. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
7 more
  1. 2024Massive Activations in Large Language Modelscited by 294
  2. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  3. 2023Microscaling Data Formats for Deep Learningcited by 152
  4. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 131
  5. 2023OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantizationcited by 92
  6. 2025TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Controlcited by 14
  7. 2026HiFloat4 Format for Language Model Inferencecited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026PRQuant: Permutation Residual Quantization for Low-Overhead Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Rethinking Channel Dimensions to Isolate Outliers for Low-bit Weight Quantization of Large Language Models
  2. 2026MosaicQuant: Inlier-Outlier Disaggregation for Unified 4-Bit LLM Quantization
  3. 2026OCGQuant: Outlier-Companion Grouping for NVFP4 Quantization
  4. 2026MUXQ: Mixed-to-Uniform Precision MatriX Quantization via Low-Rank Outlier Decomposition
  5. 2025Outlier-Safe Pre-Training for Robust 4-Bit Quantization of Large Language Models
  6. 2026OffQ: Taming Structured Outliers in LLM Quantization by Offsetting
  7. 2024OPAL: Outlier-Preserved Microscaling Quantization Accelerator for Generative Large Language Models
  8. 2026FlashQuant: Sparse-Dense Fusion for Memory-Efficient Outlier-Aware LLM Inference