mlsanityData to 7 Oct 2026

T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge

arXiv 2407.00088 ↗First posted 25 Jun 2024cs.DC cs.AIcited by 47

Not on the map. Why?

Abstract (arXiv)

The deployment of Large Language Models (LLMs) on edge devices is increasingly important to enhance on-device intelligence. Weight quantization is crucial for reducing the memory footprint of LLMs on devices. However, low-bit LLMs necessitate mixed precision matrix multiplication (mpGEMM) of low precision weights and high precision activations during inference. Existing systems, lacking native support for mpGEMM, resort to dequantize weights for high precision computation. Such an indirect way can lead to a significant inference overhead. In this paper, we introduce T-MAC, an innovative lookup table(LUT)-based method designed for efficient low-bit LLM (i.e., weight-quantized LLM) inference on CPUs. T-MAC directly supports mpGEMM without dequantization, while simultaneously eliminating multiplications and reducing additions required. Specifically, T-MAC transforms the traditional data-type-centric multiplication to bit-wise table lookup, and enables a unified and scalable mpGEMM solution. Our LUT-based kernels scale linearly to the weight bit-width. Evaluated on low-bit Llama and BitNet models, T-MAC demonstrates up to 4x increase in throughput and 70% reduction in energy consumption compared to llama.cpp. For BitNet-b1.58-3B, T-MAC delivers a token generation throughput of 30 tokens/s with a single core and 71 tokens/s with eight cores on M2-Ultra, and 11 tokens/s on lower-end devices like Raspberry Pi 5, which significantly exceeds the adult average reading speed. T-MAC with LUT-based computing paradigm, paves the way for the practical deployment of low-bit LLMs on resource-constrained edge devices without compromising computational efficiency. The system is open-sourced at https://github.com/microsoft/T-MAC .

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 17 of its 32 references that are in the map. First posted on arXiv 25 Jun 2024.

Builds on

17 of 32 references.

References · 17

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2016Pointer Sentinel Mixture Modelscited by 2,992
  3. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  4. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  5. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  6. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  7. 2024Yi: Open Foundation Models by 01.AIcited by 706
  8. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
  9. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 175
  10. 2018Learning to Optimize Tensor Programscited by 138
  11. 2024BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillationcited by 80
  12. 2024OneBit: Towards Extremely Low-bit Large Language Modelscited by 77
5 more
  1. 2021Multiplying Matrices Without Multiplyingcited by 30
  2. 2023LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookupcited by 19
  3. 2023DeepGEMM: Accelerated Ultra Low-Precision Inference on CPU Architectures using Lookup Tablescited by 8
  4. 2023Look-Up mAI GeMM: Increasing AI GeMMs Performance by Nearly 2.5x via msGeMMcited by 3
  5. 2020High Throughput Matrix-Matrix Multiplication between Asymmetric Bit-Width Operandscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 21; 2026: 21.

47 papers.

Sort:

By area:

  1. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 123
  2. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 36
  3. 2025Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latencycited by 32
  4. 2024BlueLM-V-3B: Algorithm and System Co-Design for Multimodal Large Language Models on Mobile Devicescited by 25
  5. 2025Bitnet.cpp: Efficient Edge Inference for Ternary LLMscited by 17
  6. 2024LUT Tensor Core: A Software-Hardware Co-Design for LUT-Based Low-Bit LLM Inferencecited by 14
  7. 20241-bit AI Infra: Part 1.1, Fast and Lossless BitNet b1.58 Inference on CPUscited by 9
  8. 2025TeLLMe: An Energy-Efficient Ternary LLM Accelerator for Prefilling and Decoding on Edge FPGAscited by 5
  9. 2026Sherry: Hardware-Efficient 1.25-Bit Ternary Quantization via Fine-grained Sparsificationcited by 5
  10. 2026AngelSlim: A more accessible, comprehensive, and efficient toolkit for large model compressioncited by 5
  11. 2025BTC-LLM: Efficient Sub-1-Bit LLM Quantization via Learnable Transformation and Binary Codebookcited by 4
  12. 2025LUT-LLM: Efficient Large Language Model Inference with Memory-based Computations on FPGAscited by 4
35 more
  1. 2025DynaKV: Enabling Accurate and Efficient Long-Sequence LLM Decoding on Smartphonescited by 4
  2. 2025T-SAR: A Full-Stack Co-design for CPU-Only Ternary LLM Inference via In-Place SIMD ALU Reorganizationcited by 4
  3. 2025Scaling Up On-Device LLMs via Active-Weight Swapping Between DRAM and Flashcited by 3
  4. 2025CodeGEMM: A Codebook-Centric Approach to Efficient GEMM in Quantized LLMscited by 3
  5. 2025Fairy2i: Training Complex LLMs from Real LLMs with All Parameters in ± 1, ± icited by 2
  6. 2025Binary Neural Networks for Large Language Model: A Surveycited by 1
  7. 2025LCD: Advancing Extreme Low-Bit Clustering for Large Language Models via Knowledge Distillationcited by 1
  8. 2025Pushing the Envelope of LLM Inference with Ultra-Low-Bit Quantized Modelscited by 1
  9. 2025Rethinking 1-bit Optimization Leveraging Pre-trained Large Language Modelscited by 1
  10. 2025Tiny but Mighty: A Software-Hardware Co-Design Approach for Efficient Multimodal Inference on Battery-Powered Small Devicescited by 1
  11. 2025TeLLMe v2: An Efficient End-to-End Ternary LLM Prefill and Decode Accelerator with Table-Lookup Matmul on Edge FPGAscited by 1
  12. 2025Vec-LUT: Vector Table Lookup for Parallel Ultra-Low-Bit LLM Inference on Edge Devicescited by 1
  13. 2026SFMP: Fine-Grained, Hardware-Friendly and Search-Free Mixed-Precision Quantization for Large Language Modelscited by 1
  14. 2026Breaking the 1.58-bit Barrier for Ternary LLMscited by 1
  15. 2025MagicVL-2B: Empowering Vision-Language Models on Mobile Devices with Lightweight Visual Encoders via Curriculum Learningcited by 0
  16. 2025SAIL: SRAM-Accelerated LLM Inference System with Lookup-Table-based GEMVcited by 0
  17. 2025ELUTQ: Optimizing Quantization Accuracy under LUT-Based Computation for Edge LLMscited by 0
  18. 2025BitRL-Light: 1-bit LLM Agents with Deep Reinforcement Learning for Energy-Efficient Smart Home Lighting Optimizationcited by 0
  19. 2026Vec-QMDP: Vectorized POMDP Planning on CPUs for Real-Time Autonomous Drivingcited by 0
  20. 2026pQuant: Towards Effective Low-Bit Language Models via Decoupled Linear Quantization-Aware Trainingcited by 0
  21. 2026CodeQuant: Unified Clustering and Quantization for Enhanced Outlier Smoothing in Low-Precision Mixture-of-Expertscited by 0
  22. 2026FairyFuse: Multiplication-Free LLM Inference on CPUs via Fused Ternary Kernelscited by 0
  23. 2026ViM-Q: Scalable Algorithm-Hardware Co-Design for Vision Mamba Model Inference on FPGAcited by 0
  24. 2026Litespark Inference For CPUs: Ultra-Fast SIMD Framework for Ternary (1.58-bit) Language Modelscited by 0
  25. 2026SPEAR: A System for Post-Quantization Error-Adaptive Recovery Enabling Efficient Low-Bit LLM Servingcited by 0
  26. 2026SMEPilot: Characterizing and Optimizing LLM Inference with Scalable Matrix Extensionscited by 0
  27. 2026Cache-Resident LLM Inference in GB-Scale Last-Level Cachescited by 0
  28. 2026StreamDQ: Near-Memory Weight DeQuantization in Custom HBM for Scalable AI Inference Accelerationcited by 0
  29. 2026PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inferencecited by 0
  30. 2026ExaGEMM: Exploration Framework for CPU-Driven ML Inference via Associative In-Register Computing for Low-Bit GEMMcited by 0
  31. 2026Tied Trit-Planes: Constraining PTQTP to a Uniform Nine-Level Quantizer, with a Persistent Folded Format for Disk-Streamed Mixture-of-Experts Servingcited by 0
  32. 2026Where Post-Training Quantization Breaks Text Embedders: A Measured Map Across Four Embedder Familiescited by 0
  33. 2026Empowering Hybrid Attention Models on NPUscited by 0
  34. 2026Dynamic Flow, Static Graph: KV Cache Reuse for Efficient LLM Serving on Mobile NPUscited by 0
  35. 2026On the Relationship between Model Quantization and Model Inversion Attackscited by 0