mlsanityData to 7 Oct 2026

Optimizing Inference Performance of Transformers on CPUs

arXiv 2102.06621 ↗First posted 12 Feb 2021cs.CL cs.AI cs.DCcited by 7

Not on the map. Why?

Abstract (arXiv)

The Transformer architecture revolutionized the field of natural language processing (NLP). Transformers-based models (e.g., BERT) power many important Web services, such as search, translation, question-answering, etc. While enormous research attention is paid to the training of those models, relatively little efforts are made to improve their inference performance. This paper comes to address this gap by presenting an empirical analysis of scalability and performance of inferencing a Transformer-based model on CPUs. Focusing on the highly popular BERT model, we identify key components of the Transformer architecture where the bulk of the computation happens, and propose three optimizations to speed them up. The optimizations are evaluated using the inference benchmark from HuggingFace, and are shown to achieve the speedup of up to x2.37. The considered optimizations do not require any changes to the implementation of the models nor affect their accuracy.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 18 of its 37 references that are in the map. First posted on arXiv 12 Feb 2021.

Builds on

18 of 37 references.

References · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  6. 2020Longformer: The Long-Document Transformercited by 3,362
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  8. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  9. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  10. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  11. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  12. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
6 more
  1. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  2. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71
  3. 2018Optimizing CNN Model Inference on CPUscited by 29
  4. 2020DeFormer: Decomposing Pre-trained Transformers for Faster Question Answeringcited by 29
  5. 2020Which *BERT? A Survey Organizing Contextualized Encoderscited by 26
  6. 2019Exploiting Parallelism Opportunities with Deep Learning Frameworkscited by 10

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 1; 2025: 2.

7 papers.

  1. 2024IceFormer: Accelerated Inference with Long-Sequence Transformers on CPUscited by 8
  2. 2025Scaling Laws for Energy Efficiency of Local LLMscited by 4
  3. 2023AttMEMO : Accelerating Transformers with Memoization on Big Memory Systemscited by 2
  4. 2023Packrat: Automatic Reconfiguration for Latency Minimization in CPU-based DNN Servingcited by 2
  5. 2025Energy-Efficient Transformer Inference: Optimization Strategies for Time Series Classificationcited by 2
  6. 2023YFlows: Systematic Dataflow Exploration and Code Generation for Efficient Neural Network Inference using SIMD Architectures on CPUscited by 1
  7. 2023Improving Inference Performance of Machine Learning with the Divide-and-Conquer Principlecited by 0