# Vec-LUT: Vector Table Lookup for Parallel Ultra-Low-Bit LLM Inference on Edge Devices

arXiv: https://arxiv.org/abs/2512.06443 · HTML: https://mlsanity.com/paper/2512.06443/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 52 references that are in the map.
- First posted on arXiv 6 Dec 2025.

## Builds on (most cited first, 22 of 22)

- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits](https://mlsanity.com/paper/2402.17764/) (2024)
- [AutoDroid: LLM-powered Task Automation in Android](https://mlsanity.com/paper/2308.15272/) (2023)
- [EfficientQAT: Efficient Quantization-Aware Training for Large Language Models](https://mlsanity.com/paper/2407.11062/) (2024)
- [BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillation](https://mlsanity.com/paper/2402.10631/) (2024)
- [T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge](https://mlsanity.com/paper/2407.00088/) (2024)
- [CodeMonkeys: Scaling Test-Time Compute for Software Engineering](https://mlsanity.com/paper/2501.14723/) (2025)
- [ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization](https://mlsanity.com/paper/2502.02631/) (2025)
- [StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognition](https://mlsanity.com/paper/2503.06220/) (2025)
- [BitNet b1.58 2B4T Technical Report](https://mlsanity.com/paper/2504.12285/) (2025)
- [Parallel Scaling Law for Language Models](https://mlsanity.com/paper/2505.10475/) (2025)
- [Mobile Foundation Model as Firmware](https://mlsanity.com/paper/2308.14363/) (2023)
- [Bitnet.cpp: Efficient Edge Inference for Ternary LLMs](https://mlsanity.com/paper/2502.11880/) (2025)
- [Spectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scale](https://mlsanity.com/paper/2407.12327/) (2024)
- [LUT Tensor Core: A Software-Hardware Co-Design for LUT-Based Low-Bit LLM Inference](https://mlsanity.com/paper/2408.06003/) (2024)
- [AutoIOT: LLM-Driven Automated Natural Language Programming for AIoT Applications](https://mlsanity.com/paper/2503.05346/) (2025)
- [EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devices](https://mlsanity.com/paper/2507.01438/) (2025)
- [TENET: An Efficient Sparsity-Aware LUT-Centric Architecture for Ternary LLM Inference On Edge](https://mlsanity.com/paper/2509.13765/) (2025)
- [T-SAR: A Full-Stack Co-design for CPU-Only Ternary LLM Inference via In-Place SIMD ALU Reorganization](https://mlsanity.com/paper/2511.13676/) (2025)

## Built on by (most cited first, 2 of 2)

- [ExaGEMM: Exploration Framework for CPU-Driven ML Inference via Associative In-Register Computing for Low-Bit GEMM](https://mlsanity.com/paper/2607.14622/) (2026)
- [UnionSparse: An Index-Efficient Sparsity Framework for Low-Bit Sparse LLM Inference on Edge](https://mlsanity.com/paper/2608.09291/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly deployed on edge devices. To meet strict resource constraints, real-world deployment has pushed LLM quantization from 8-bit to 4-bit, 2-bit, and now 1.58-bit. Combined with lookup table (LUT)-based inference, CPUs run these ultra-low-bit LLMs even faster than NPUs, opening new opportunities for ubiquitous on-device intelligence. However, this paper identifies that LUT-based inference underutilizes memory bandwidth during parallel inference, which is required for prefilling, test-time scaling, and other multi-token scenarios. The root cause is the scalar LUT paradigm, which performs repetitive and non-contiguous memory accesses for each token. To solve the issue, we propose vector LUT, a new lookup paradigm that constructs a unified LUT across parallel tokens, and performs a single 1 rightarrow N lookup per index. To realize it efficiently, we further introduce (1) Vector LUT-Centric Tensor Layout, and (2) Cache-Aware Streamed Lookup techniques. Evaluations on 5 edge devices across 3 LLMs show that Vec-LUT outperforms state-of-the-art baselines by up to 4.2×. Our implementation is integrated into llama.cpp. The code is available at https://github.com/OpenBitSys/vlut.cpp.