# Highly Optimized Kernels and Fine-Grained Codebooks for LLM Inference on Arm CPUs

arXiv: https://arxiv.org/abs/2501.00032 · HTML: https://mlsanity.com/paper/2501.00032/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Dec 2024.

## Built on by (most cited first, 5 of 5)

- [MusaCoder: Native GPU Kernel Generation with Full-Stack Training on Moore Threads GPU](https://mlsanity.com/paper/2606.04847/) (2026)
- [AttestLLM: Efficient Attestation Framework for Billion-scale On-device LLMs](https://mlsanity.com/paper/2509.06326/) (2025)
- [From Muscle to Text with MyoText: sEMG to Text via Finger Classification and Transformer-Based Decoding](https://mlsanity.com/paper/2601.03098/) (2026)
- [A Quantized Native Runtime for On-Device Semantic Audio Generation](https://mlsanity.com/paper/2607.08526/) (2026)
- [Llama-Mobile: Efficient 2.7-Bit Quantization of VLMs](https://mlsanity.com/paper/2608.21134/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have transformed the way we think about language understanding and generation, enthralling both researchers and developers. However, deploying LLMs for inference has been a significant challenge due to their unprecedented size and resource requirements. While quantizing model weights to sub-byte precision has emerged as a promising solution to ease memory pressure, the group quantization formats commonly used for LLM quantization have significant compute overheads and a resource-intensive dequantization process. As a result, a higher proportion of compute instructions do not perform multiplies, i.e., real work, rendering them unsuitable for meeting the required latency requirements for LLMs deployed on commodity CPUs. In this work, we propose a set of highly optimized kernels to accelerate LLM inference and unleash the full potential of CPUs, particularly Arm CPUs. These kernels amortize the cost of loading the operands and the cost of weight unpacking across multiple output rows. This, along with the introduction of an optimized interleaved group data layout for weights and decompression path optimizations to reduce unnecessary operations and dequantization overhead while maximizing the use of vector and matrix multiply operations, significantly improves the efficiency of MAC operations. Furthermore, we present a groupwise non-uniform codebook-based quantization method for ultra-low-precision quantization of LLMs to better match non-uniform patterns in their weight distributions, demonstrating better throughput during token generation while ensuring better quality than the state-of-the-art. Applying these improvements to 4-bit LLMs results in a 3-3.2x improvement in prompt processing and a 2x improvement in autoregressive decoding on Arm CPUs, compared to LLaMA.cpp-based solution. The optimized kernels are available at https://github.com/ggerganov/llama.cpp.