mlsanityData to 7 Oct 2026

NoMAD-Attention: Efficient LLM Inference on CPUs Through Multiply-add-free Attention

arXiv 2403.01273 ↗First posted 2 Mar 2024cs.LG cs.AI cs.CLcited by 12

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large language model inference on Central Processing Units (CPU) is challenging due to the vast quantities of expensive Multiply-Add (MAD) matrix operations in the attention computations. In this paper, we argue that there is a rare gem in modern CPUs, Single-Instruction-Multiple-Data (SIMD) registers, which allow for ultra-low-latency lookups in batch. We leverage this unique capability of CPUs to propose NoMAD-Attention, an efficient attention algorithm that replaces MAD operations with in-register lookups. Through hardware-aware algorithmic designs, NoMAD-Attention achieves the computation of attention scores using repeated fast accesses to SIMD registers despite their highly limited sizes. Moreover, NoMAD-Attention works with pre-trained attention-based LLMs without model finetuning. Empirical evaluations demonstrate that NoMAD-Attention maintains the quality of the original LLMs well, and speeds up the 4-bit quantized LLaMA-7B-based model by up to 2× at 16k context length. Our results are reproducible at https://github.com/tonyzhang617/nomad-dist.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 39 of its 66 references that are in the map. First posted on arXiv 2 Mar 2024.

Builds on

39 of 66 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2016Pointer Sentinel Mixture Modelscited by 2,992
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  11. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  12. 2022Emergent Abilities of Large Language Modelscited by 2,483
26 more
  1. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  2. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  3. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  4. 2020Reformer: The Efficient Transformercited by 1,658
  5. 2020Rethinking Attention with Performerscited by 1,584
  6. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  7. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  8. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  9. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  10. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  11. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  12. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 464
  13. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  14. 2023Challenges and Applications of Large Language Modelscited by 251
  15. 2021Scatterbrain: Unifying Sparse and Low-rank Attention Approximationcited by 128
  16. 2021Lawformer: A Pre-trained Language Model for Chinese Legal Long Documentscited by 115
  17. 2023Pushing Large Language Models to the 6G Edge: Vision, Challenges, and Opportunitiescited by 89
  18. 2023HyperAttention: Long-context Attention in Near-Linear Timecited by 85
  19. 2019SLIDE : In Defense of Smart Algorithms over Hardware Acceleration for Large-Scale Deep Learning Systemscited by 46
  20. 2021Multiplying Matrices Without Multiplyingcited by 30
  21. 2023Efficient LLM Inference on CPUscited by 24
  22. 2016Scalable and Sustainable Deep Learning via Randomized Hashingcited by 17
  23. 2023A Hardware Evaluation Framework for Large Language Model Inferencecited by 14
  24. 2019Summarizing CPU and GPU Design Trends with Product Datacited by 10
  25. 2017Accelerated Nearest Neighbor Search with Quick ADCcited by 9
  26. 2023PQA: Exploring the Potential of Product Quantization in DNN Hardware Accelerationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 4; 2026: 3.

12 papers.

  1. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantizationcited by 74
  2. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 36
  3. 202570% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11)cited by 23
  4. 2024LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Gridcited by 20
  5. 2024Sketch to Adapt: Fine-Tunable Sketches for Efficient LLM Adaptationcited by 3
  6. 2024Towards a Middleware for Large Language Modelscited by 2
  7. 2025Memory Access Characterization of Large Language Models in CPU Environment and its Potential Impactscited by 1
  8. 2025SAIL: SRAM-Accelerated LLM Inference System with Lookup-Table-based GEMVcited by 0
  9. 2025BitRL-Light: 1-bit LLM Agents with Deep Reinforcement Learning for Energy-Efficient Smart Home Lighting Optimizationcited by 0
  10. 2026ExaGEMM: Exploration Framework for CPU-Driven ML Inference via Associative In-Register Computing for Low-Bit GEMMcited by 0
  11. 2026Gradient-free Task-Conditioned Retrieval for On-Device In-Context Learningcited by 0
  12. 2026SCOPE: Subspace Clustering with Online Per-Head Top-K Estimation for Sparse Video Attentioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LMDeploy Accelerates Mixed-Precision LLM Inference with TurboMind
  2. 2024CHAI: Clustered Head Attention for Efficient LLM Inference
  3. 2025Glinthawk: A Two-Tiered Architecture for Offline LLM Inference
  4. 2023SparQ Attention: Bandwidth-Efficient LLM Inference
  5. 2025Reimagining Memory Access for LLM Inference: Compression-Aware Memory Controller Design
  6. 2025HGCA: Hybrid GPU-CPU Attention for Long Context LLM Inference
  7. 2026PackInfer: Compute- and I/O-Efficient Attention for Batched LLM Inference
  8. 2025HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading