mlsanityData to 7 Oct 2026

ELUTQ: Optimizing Quantization Accuracy under LUT-Based Computation for Edge LLMs

arXiv 2510.19482 ↗First posted 22 Oct 2025cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Weight quantization effectively reduces memory consumption and enable the deployment of Large Language Models on edge devices, yet existing hardware-friendly methods often rely on uniform quantization, which suffers from poor weight-distribution fitting and high dequantization overhead under low-bit settings. In this paper, we propose ELUTQ, an efficient quantization framework featuring a novel quantization format termed Hierarchical Linear Quantization (HLQ). HLQ is designed to better capture the statistical characteristics of weights and eliminate dequantization overhead using Bit-serial LUT-based GEMM operations. HLQ significantly improves model accuracy under low-bit settings and achieves performance comparable to QAT methods without any retraining of the weights. Moreover, an optimized quantization pipeline is integrated into ELUTQ, enabling it to complete the quantization of LLaMA 3.1-70B using only 64 GB of CPU memory and 48 GB of VRAM, reducing the hardware requirements for large-scale model quantization. To enable efficient deployment on edge devices, ELUTQ designs high-performance kernels to support end-to-end inference. Our 2-bit LLaMA3.1-8B achieves 1.5x speedup over AWQ on RTX 3090. Code is available at https://github.com/Nkniexin/ELUTQ.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 32 of its 37 references that are in the map. First posted on arXiv 22 Oct 2025.

Builds on

32 of 37 references.

References · 32

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  8. 2016Pointer Sentinel Mixture Modelscited by 2,992
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  10. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  11. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  12. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
20 more
  1. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  2. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  3. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
  4. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  5. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 223
  6. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 195
  7. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 154
  8. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 123
  9. 2022LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Modelscited by 98
  10. 2023PB-LLM: Partially Binarized Large Language Modelscited by 80
  11. 2021Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimationcited by 78
  12. 2024Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMscited by 50
  13. 2024T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edgecited by 47
  14. 2024DB-LLM: Accurate Dual-Binarization for Efficient LLMscited by 35
  15. 2018Alternating Multi-bit Quantization for Recurrent Neural Networkscited by 30
  16. 2024ApiQ: Finetuning of 2-Bit Quantized Large Language Modelcited by 27
  17. 2024ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterizationcited by 27
  18. 2025FIGLUT: An Energy-Efficient Accelerator Design for FP-INT GEMM Using Look-Up Tablescited by 8
  19. 2025AnyBCQ: Hardware Efficient Flexible Binary-Coded Quantization for Multi-Precision LLMscited by 8
  20. 2025GANQ: GPU-Adaptive Non-Uniform Quantization for Large Language Modelscited by 3

Built on by

Not cited yet.