mlsanityData to 7 Oct 2026

OASIS: Outlier-Aware LUT-Based GEMM with Dual-Side Quantization for LLM Inference Acceleration

arXiv 2507.23035 ↗First posted 30 Jul 2025cs.LG cs.ARcited by 2

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

Large language models (LLMs) have demonstrated impressive capabilities across a wide range of applications, but demand substantial memory and compute resources during inference. Existing quantization methods expose a trade-off between efficiency and accuracy: weight-only quantization (WOQ) incurs costly dequantization overheads, while integer weight-and-activation quantization (INT-WAQ) reduces precision and degrades model quality. Non-uniform weight-and-activation quantization (NU-WAQ) can better capture the non-uniform distributions of LLM weights and activations, yet remains incompatible with conventional low-precision compute units. This paper presents OASIS, a lookup table (LUT)-based architecture that enables efficient general matrix multiplication (GEMM) between non-uniformly quantized weights and activations without requiring dequantization. OASIS employs pre-computed Cartesian Product LUTs, achieving a 64x reduction in LUT size and enabling a 1024x higher computational parallelism over existing LUT-based GEMM methods. To preserve accuracy under aggressive activation quantization, OASIS introduces an outlier-aware quantization scheme with concurrent LUT-based GEMM and error compensation for outliers. Furthermore, we design Orizuru, an efficient top-k detection engine for real-time activation outlier identification. According to extensive evaluations, OASIS incurs an average accuracy drop of only 1.98% compared to the FP16 baseline, which is 5.18% lower than Atom. On the hardware side, OASIS achieves an average 3.00x speedup and a 1.44x energy efficiency improvement compared to the FIGLUT accelerator.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 43 of its 63 references that are in the map. First posted on arXiv 30 Jul 2025.

Builds on

43 of 63 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024SpinQuant: LLM quantization with learned rotations26 May
  2. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving7 May
  3. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs30 Mar
  4. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization31 Jan
  5. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Serving29 Oct

Other references · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2023Mistral 7Bcited by 3,065
  12. 2016Pointer Sentinel Mixture Modelscited by 2,992
26 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  2. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  3. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  5. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  6. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  7. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  8. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  9. 2018LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networkscited by 272
  10. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 228
  11. 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 194
  12. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 165
  13. 2023ChatEDA: A Large Language Model Powered Autonomous Agent for EDAcited by 118
  14. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 116
  15. 2022LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Modelscited by 98
  16. 2024Scaling Laws for Precisioncited by 92
  17. 2023LLM-FP4: 4-Bit Floating-Point Quantized Transformerscited by 65
  18. 2023GPT4AIGChip: Towards Next-Generation AI Accelerator Design Automation via Large Language Modelscited by 59
  19. 2025Training LLMs with MXFP4cited by 36
  20. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 21
  21. 2024LUT Tensor Core: A Software-Hardware Co-Design for LUT-Based Low-Bit LLM Inferencecited by 14
  22. 2023Flexagon: A Multi-Dataflow Sparse-Sparse Matrix Multiplication Accelerator for Efficient DNN Processingcited by 13
  23. 2023Fast and Efficient 2-bit LLM Inference on GPU: 2/4/16-bit in a Weight Matrix with Asynchronous Dequantizationcited by 13
  24. 2025FIGLUT: An Energy-Efficient Accelerator Design for FP-INT GEMM Using Look-Up Tablescited by 8
  25. 2024Crystal: Illuminating LLM Abilities on Language and Codecited by 5
  26. 2023Block-Wise Mixed-Precision Quantization: Enabling High Efficiency for Practical ReRAM-based DNN Acceleratorscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Retrieval-Augmented Detection of Potentially Abusive Clauses in Chilean Terms of Servicecited by 0
  2. 2026AIR-LLM: Broadcasting AI Weights over Radio for Memory-Free Edge LLM Inference via RF Computingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Multi-Scale Dequant: Eliminating Dequantization Bottleneck via Activation Decomposition for Efficient LLM Inference
  2. 2025Vec-LUT: Vector Table Lookup for Parallel Ultra-Low-Bit LLM Inference on Edge Devices
  3. 2023SmoothQuant+: Accurate and Efficient 4-bit Post-Training WeightQuantization for LLM
  4. 2023FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMs
  5. 2024MixPE: Quantization and Hardware Co-design for Efficient LLM Inference
  6. 2023Efficient LLM Inference on CPUs
  7. 2024Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Format
  8. 2025LiquidGEMM: Hardware-Efficient W4A8 GEMM Kernel for High-Performance LLM Serving