mlsanityData to 14 Oct 2026

Efficient LLM Inference on CPUs

arXiv 2311.00502 ↗First posted 1 Nov 2023cs.LG cs.AI cs.CLcited by 24

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable performance and tremendous potential across a wide range of tasks. However, deploying these models has been challenging due to the astronomical amount of model parameters, which requires a demand for large memory capacity and high memory bandwidth. In this paper, we propose an effective approach that can make the deployment of LLMs more efficiently. We support an automatic INT4 weight-only quantization flow and design a special LLM runtime with highly-optimized kernels to accelerate the LLM inference on CPUs. We demonstrate the general applicability of our approach on popular LLMs including Llama2, Llama, GPT-NeoX, and showcase the extreme inference efficiency on CPUs. The code is publicly available at: https://github.com/intel/intel-extension-for-transformers.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 23 of its 29 references that are in the map. First posted on arXiv 1 Nov 2023.

Builds on

23 of 29 references.

References · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  9. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  10. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  11. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  12. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
11 more
  1. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  2. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  3. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  4. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  5. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  6. 2022FP8 Formats for Deep Learningcited by 196
  7. 2022FP8 Quantization: The Power of the Exponentcited by 88
  8. 2023Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMscited by 50
  9. 2023ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formatscited by 44
  10. 2023Efficient Post-training Quantization with FP8 Formatscited by 29
  11. 2023TEQ: Trainable Equivalent Transformation for Quantization of LLMscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 8; 2025: 10; 2026: 3.

24 papers.

Sort:

By area:

  1. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  2. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 38
  3. 2023Scaling Down to Scale Up: A Cost-Benefit Analysis of Replacing OpenAI's LLM with Open Source SLMs in Productioncited by 21
  4. 2025PAPI: Exploiting Dynamic Parallelism in Large Language Model Decoding with a Processing-In-Memory-Enabled Computing Systemcited by 20
  5. 2025A Cost-Benefit Analysis of On-Premise Large Language Model Deployment: Breaking Even with Commercial LLM Servicescited by 17
  6. 2024Accelerating Retrieval-Augmented Generationcited by 14
  7. 2024NoMAD-Attention: Efficient LLM Inference on CPUs Through Multiply-add-free Attentioncited by 12
  8. 2025ML-SpecQD: Multi-Level Speculative Decoding with Quantized Draftscited by 9
  9. 2023Generative AI Beyond LLMs: System Implications of Multi-Modal Generationcited by 6
  10. 2025Method-Based Reasoning for Large Language Models: Extraction, Reuse, and Continuous Improvementcited by 6
  11. 2025Mixed-Precision Quantization for Language Models: Techniques and Prospectscited by 5
  12. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
12 more
  1. 2024Towards Enabling FAIR Dataspaces Using Large Language Modelscited by 2
  2. 2024FEDSTR: Money-In AI-Out | A Decentralized Marketplace for Federated Learning and LLM Training on the NOSTR Protocolcited by 2
  3. 2024Inference Acceleration for Large Language Models on CPUscited by 2
  4. 2025Small Language Models (SLMs) Can Still Pack a Punch: A survey (updated 2026)cited by 2
  5. 2025EdgeProfiler: A Fast Profiling Framework for Lightweight LLMs on Edge Using Analytical Modelcited by 2
  6. 2023Efficient Key-Based Adversarial Defense for ImageNet by Using Pre-trained Modelcited by 1
  7. 2025SAIL: SRAM-Accelerated LLM Inference System with Lookup-Table-based GEMVcited by 1
  8. 2026Efficient and Privacy Aware Edge Cloud Collaborative Inference for Large Language Modelscited by 1
  9. 2025V-Seek: Accelerating LLM Reasoning on Open-hardware Server-class RISC-V Platformscited by 0
  10. 2025Striking the Right Balance between Compute and Copy: Improving LLM Inferencing Under Speculative Decodingcited by 0
  11. 2026Active Imitation Learning for Thermal- and Kernel-Aware LFM Inference on 3D S-NUCA Many-Corescited by 0
  12. 2026RIS-Kernel: A Model-Agnostic Architecture for Long-Context LLM Inference via Sparse Attentioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Highly Optimized Kernels and Fine-Grained Codebooks for LLM Inference on Arm CPUs
  2. 2024LLM Inference Unveiled: Survey and Roofline Model Insights
  3. 2023Efficient LLM inference solution on Intel GPU
  4. 2024GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments
  5. 2025TokenSim: Enabling Hardware and Software Exploration for Large Language Model Inference Systems
  6. 2024Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Format
  7. 2023Dissecting the Runtime Performance of the Training, Fine-tuning, and Inference of Large Language Models
  8. 2025LUT-LLM: Efficient Large Language Model Inference with Memory-based Computations on FPGAs