mlsanityData to 7 Oct 2026

LUT Tensor Core: A Software-Hardware Co-Design for LUT-Based Low-Bit LLM Inference

arXiv 2408.06003 ↗First posted 12 Aug 2024cs.AR cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

Large Language Model (LLM) inference becomes resource-intensive, prompting a shift toward low-bit model weights to reduce the memory footprint and improve efficiency. Such low-bit LLMs necessitate the mixed-precision matrix multiplication (mpGEMM), an important yet underexplored operation involving the multiplication of lower-precision weights with higher-precision activations. Off-the-shelf hardware does not support this operation natively, leading to indirect, thus inefficient, dequantization-based implementations. In this paper, we study the lookup table (LUT)-based approach for mpGEMM and find that a conventional LUT implementation fails to achieve the promised gains. To unlock the full potential of LUT-based mpGEMM, we propose LUT Tensor Core, a software-hardware co-design for low-bit LLM inference. LUT Tensor Core differentiates itself from conventional LUT designs through: 1) software-based optimizations to minimize table precompute overhead and weight reinterpretation to reduce table storage; 2) a LUT-based Tensor Core hardware design with an elongated tiling shape to maximize table reuse and a bit-serial design to support diverse precision combinations in mpGEMM; 3) a new instruction set and compilation optimizations for LUT-based mpGEMM. LUT Tensor Core significantly outperforms existing pure software LUT implementations and achieves a 1.44× improvement in compute density and energy efficiency compared to previous state-of-the-art LUT-based accelerators.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 32 of its 80 references that are in the map. First posted on arXiv 12 Aug 2024.

Builds on

32 of 80 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  7. 2016Pointer Sentinel Mixture Modelscited by 2,992
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  10. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  11. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  12. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
20 more
  1. 2024Yi: Open Foundation Models by 01.AIcited by 706
  2. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  3. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  4. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  5. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  6. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  7. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  8. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
  9. 2020Ansor: Generating High-Performance Tensor Programs for Deep Learningcited by 199
  10. 2023Microscaling Data Formats for Deep Learningcited by 132
  11. 2020GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inferencecited by 103
  12. 2022LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Modelscited by 98
  13. 2023OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantizationcited by 83
  14. 2024BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillationcited by 80
  15. 2023Training Transformers with 4-bit Integerscited by 58
  16. 2024T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edgecited by 47
  17. 2022ANT: Exploiting Adaptive Numerical Data Type for Low-bit Deep Neural Network Quantizationcited by 42
  18. 2024Spectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scalecited by 20
  19. 2020BiQGEMM: Matrix Multiplication with Lookup Table For Binary-Coding-based Quantized DNNscited by 17
  20. 2022Mokey: Enabling Narrow Fixed-Point Inference for Out-of-the-Box Floating-Point Transformer Modelscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 8.

14 papers.

Sort:

By area:

  1. 2025LayerScope: Predictive Cross-Layer Scheduling for Efficient Multi-Batch MoE Inference on Legacy Serverscited by 8
  2. 2025V-Rex: Real-Time Streaming Video LLM Acceleration via Dynamic KV Cache Retrievalcited by 4
  3. 2025P3-LLM: An Integrated NPU-PIM Accelerator for Edge LLM Inference Using Hybrid Numerical Formatscited by 3
  4. 2025OASIS: Outlier-Aware LUT-Based GEMM with Dual-Side Quantization for LLM Inference Accelerationcited by 2
  5. 2025Automating Hardware Design and Verification from Architectural Papers via a Neural-Symbolic Graph Frameworkcited by 1
  6. 2025Vec-LUT: Vector Table Lookup for Parallel Ultra-Low-Bit LLM Inference on Edge Devicescited by 1
  7. 2026LLM Inference in a Flash!cited by 1
  8. 2026CodeQuant: Unified Clustering and Quantization for Enhanced Outlier Smoothing in Low-Precision Mixture-of-Expertscited by 0
  9. 2026ViM-Q: Scalable Algorithm-Hardware Co-Design for Vision Mamba Model Inference on FPGAcited by 0
  10. 2026StreamDQ: Near-Memory Weight DeQuantization in Custom HBM for Scalable AI Inference Accelerationcited by 0
  11. 2026DeVIT: Low-Power Vision Transformer Acceleration Using Delta Computationcited by 0
  12. 2026FluxBin: Flexible LUT-based Ultra-low-bit LLM Inference by Algorithm-Kernel Synergycited by 0
2 more
  1. 2026Vortex: Bridging Extreme Compression and Efficient LLM Inferencecited by 0
  2. 2026AIR-LLM: Broadcasting AI Weights over Radio for Memory-Free Edge LLM Inference via RF Computingcited by 0