# ExaGEMM: Exploration Framework for CPU-Driven ML Inference via Associative In-Register Computing for Low-Bit GEMM

arXiv: https://arxiv.org/abs/2607.14622 · HTML: https://mlsanity.com/paper/2607.14622/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 24 references that are in the map.
- First posted on arXiv 16 Jul 2026.

## Builds on (most cited first, 13 of 13)

- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Atom: Low-bit Quantization for Efficient and Accurate LLM Serving](https://mlsanity.com/paper/2310.19102/) (2023)
- [T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge](https://mlsanity.com/paper/2407.00088/) (2024)
- [Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Models](https://mlsanity.com/paper/2503.06027/) (2025)
- [VAQF: Fully Automatic Software-Hardware Co-Design Framework for Low-Bit Vision Transformer](https://mlsanity.com/paper/2201.06618/) (2022)
- [Layer-Wise Quantization: A Pragmatic and Effective Method for Quantizing LLMs Beyond Integer Bit-Levels](https://mlsanity.com/paper/2406.17415/) (2024)
- [AMQ: Enabling AutoML for Mixed-precision Weight-Only Quantization of Large Language Models](https://mlsanity.com/paper/2509.12019/) (2025)
- [NoMAD-Attention: Efficient LLM Inference on CPUs Through Multiply-add-free Attention](https://mlsanity.com/paper/2403.01273/) (2024)
- [LiquidGEMM: Hardware-Efficient W4A8 GEMM Kernel for High-Performance LLM Serving](https://mlsanity.com/paper/2509.01229/) (2025)
- [T-SAR: A Full-Stack Co-design for CPU-Only Ternary LLM Inference via In-Place SIMD ALU Reorganization](https://mlsanity.com/paper/2511.13676/) (2025)
- [SIMD Everywhere Optimization from ARM NEON to RISC-V Vector Extensions](https://mlsanity.com/paper/2309.16509/) (2023)
- [W4A16 Mixed-Precision Matrix Multiplication on Decoupled Architecture: Kernel Design and Memory Bottleneck Analysis for Ascend NPUs](https://mlsanity.com/paper/2601.16536/) (2026)
- [Vec-LUT: Vector Table Lookup for Parallel Ultra-Low-Bit LLM Inference on Edge Devices](https://mlsanity.com/paper/2512.06443/) (2025)

## Abstract (arXiv)

Low-bit GEMM is increasingly central to efficient ML inference, yet very-low-bit execution remains a poor fit for conventional CPUs. Practical deployment spans fragmented regimes-from 1/2/4-bit weights to varying activation precision-whose feasibility, reuse opportunity, and support cost differ under fixed SIMD and register-file budgets, making lightweight CPU support selection a first-class design problem. We present ExaGEMM, a workload-aware codesign and exploration framework for CPU-native low-bit GEMM via register-resident LUT execution. The key insight is that existing SIMD datapaths already cover table generation and accumulation; the only new hardware is an in-register select/feed mechanism with explicitly modeled cost. ExaGEMM co-explores parameterized kernels and lightweight SIMD ISA support using analytical models of register feasibility, compute cost, memory traffic, and hardware overhead, pruning the candidate space by 99.2% before simulation. It then identifies non-dominated support points and generates ISA specs, gem5 patches, and GEMM kernels for validation. Across representative ML models and CPU targets, ExaGEMM improves latency by 13.29x over software-only baselines, while showing that workload-aware frontier selection is especially important for mixed-precision LLM workloads.