mlsanityData to 14 Oct 2026

PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference

arXiv 2607.14618 ↗First posted 16 Jul 2026cs.LG cs.AR cs.OScited by 0

Not on the map. Why?

Abstract (arXiv)

CPUs are the most universal target for on-device LLM inference, but existing low-bit quantization methods offer either coarse operating points or fine-grained mixed precision that is difficult to execute efficiently on CPUs. We present PolyQ, a CPU-oriented compiler/quantization co-design for activation-aware channel-wise bit allocation under a user-specified average-bit budget. PolyQ assigns per-channel bit-widths from 2,3,4,8,16, then uses a compile-time model compiler to permute and cluster channels into bit-homogeneous blocks, generate SIMD- and LUT-compatible kernels, and merge compatible permutations across operators to keep layout regularization off the runtime path. This turns fine-grained budget fitting into a practical fractional-bit deployment method for CPU-only inference. Across Falcon-H1-3B, Llama2-13B, and Qwen3-32B on WikiText-2, PolyQ provides stable quality scaling from 3--6,b and improves perplexity by 2.4--32.1% over prior methods at a 3,b target. End-to-end measurements on three representative CPUs -- workstation, laptop, and mobile -- show that compiler layout regularization reduces activation reorder traffic by up to 70.8%, prefill latency and decode throughput scale nearly proportionally with the configured bit budget, and energy/token overhead stays below 2% relative to an optimized LUT-based back-end. These results show that fractional-bit CPU deployment is practical, predictable, and energy-efficient across diverse edge targets.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 17 of its 31 references that are in the map. First posted on arXiv 16 Jul 2026.

Builds on

17 of 31 references.

References · 17

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  4. 2016Pointer Sentinel Mixture Modelscited by 3,096
  5. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  6. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  7. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 443
  8. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  9. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 177
  10. 2024A Review on Edge Large Language Models: Design, Execution, and Applicationscited by 100
  11. 2024Fast On-device LLM Inference with NPUscited by 62
  12. 2024T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edgecited by 55
5 more
  1. 2025Bitnet.cpp: Efficient Edge Inference for Ternary LLMscited by 22
  2. 2025Ecco: Improving Memory Bandwidth and Capacity for LLMs via Entropy-aware Cache Compressioncited by 14
  3. 2025AMQ: Enabling AutoML for Mixed-precision Weight-Only Quantization of Large Language Modelscited by 14
  4. 2025T-SAR: A Full-Stack Co-design for CPU-Only Ternary LLM Inference via In-Place SIMD ALU Reorganizationcited by 4
  5. 2024GCV-Turbo: End-to-end Acceleration of GNN-based Computer Vision Tasks on FPGAcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025CodeGEMM: A Codebook-Centric Approach to Efficient GEMM in Quantized LLMs
  2. 2026HBQ: Hierarchical Scaling Block Quantization with Hardware-Efficiency-Aware Design for Accurate LLM Inference
  3. 2026TileFuse: A Fused Mixed-Precision Kernel Library for Efficient Quantized LLM Inference on AMD NPUs
  4. 2026From Bits to Chips: An LLM-based Hardware-Aware Quantization Agent for Streamlined Deployment of LLMs
  5. 2025Pushing the Envelope of LLM Inference with Ultra-Low-Bit Quantized Models
  6. 2025VQ-LLM: High-performance Code Generation for Vector Quantization Augmented LLM Inference
  7. 2024LLMEasyQuant: Scalable Quantization for Parallel and Distributed LLM Inference
  8. 2026MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference