mlsanityData to 7 Oct 2026

A Survey of Techniques for Optimizing Transformer Inference

arXiv 2307.07982 ↗First posted 16 Jul 2023cs.LG cs.AR cs.CLcited by 35

Not on the map. Why?

Abstract (arXiv)

Recent years have seen a phenomenal rise in performance and applications of transformer neural networks. The family of transformer networks, including Bidirectional Encoder Representations from Transformer (BERT), Generative Pretrained Transformer (GPT) and Vision Transformer (ViT), have shown their effectiveness across Natural Language Processing (NLP) and Computer Vision (CV) domains. Transformer-based networks such as ChatGPT have impacted the lives of common men. However, the quest for high predictive performance has led to an exponential increase in transformers' memory and compute footprint. Researchers have proposed techniques to optimize transformer inference at all levels of abstraction. This paper presents a comprehensive survey of techniques for optimizing the inference phase of transformer networks. We survey techniques such as knowledge distillation, pruning, quantization, neural architecture search and lightweight network design at the algorithmic level. We further review hardware-level optimization techniques and the design of novel hardware accelerators for transformers. We summarize the quantitative results on the number of parameters/FLOPs and accuracy of several models/techniques to showcase the tradeoff exercised by them. We also outline future directions in this rapidly evolving field of research. We believe that this survey will educate both novice and seasoned researchers and also spark a plethora of research efforts in this field.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 207 of its 272 references that are in the map. First posted on arXiv 16 Jul 2023.

Builds on

207 of 272 references.

References · 207

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  9. 2020End-to-End Object Detection with Transformerscited by 7,384
  10. 2016Layer Normalizationcited by 5,579
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  12. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
38 more
  1. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  4. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  8. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  11. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  12. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  13. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  14. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  15. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  16. 2019Searching for MobileNetV3cited by 1,984
  17. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  18. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  19. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  20. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  21. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  22. 2020Reformer: The Efficient Transformercited by 1,658
  23. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  24. 2020Rethinking Attention with Performerscited by 1,584
  25. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  26. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  27. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  28. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  29. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  30. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  31. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  32. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  33. 2021Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentationcited by 1,218
  34. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  35. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  36. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  37. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  38. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090

50 most cited of 207 All 207 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 14; 2025: 12; 2026: 8.

35 papers.

Sort:

By area:

  1. 2024QuEST: Low-bit Diffusion Model Quantization via Efficient Selective Finetuningcited by 48
  2. 2024Accurate LoRA-Finetuning Quantization of LLMs via Information Retentioncited by 40
  3. 2024LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Acceleratorscited by 31
  4. 2024MPQ-DM: Mixed Precision Quantization for Extremely Low Bit Diffusion Modelscited by 20
  5. 2024Memory Is All You Need: An Overview of Compute-in-Memory Architectures for Accelerating Large Language Model Inferencecited by 14
  6. 2024Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM modelscited by 12
  7. 2025A Survey on Inference Engines for Large Language Models: Perspectives on Optimization and Efficiencycited by 12
  8. 2025MPQ-DMv2: Flexible Residual Mixed Precision Quantization for Low-Bit Diffusion Models with Temporal Distillationcited by 12
  9. 2023The Languini Kitchen: Enabling Language Modelling Research at Different Scales of Computecited by 10
  10. 2024Inference Optimization of Foundation Models on AI Acceleratorscited by 9
  11. 2025Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Servingcited by 6
  12. 2024Accelerating ViT Inference on FPGA through Static and Dynamic Pruningcited by 5
23 more
  1. 2024Optimising TinyML with Quantization and Distillation of Transformer and Mamba Models for Indoor Localisation on Edge Devicescited by 5
  2. 2025A Proximal Operator for Inducing 2:4-Sparsitycited by 4
  3. 2024LLM-Rank: A Graph Theoretical Approach to Pruning Large Language Modelscited by 2
  4. 2025Energy-Efficient Transformer Inference: Optimization Strategies for Time Series Classificationcited by 2
  5. 2025From Large-scale Audio Tagging to Real-Time Explainable Emergency Vehicle Sirens Detectioncited by 2
  6. 2026Prune-then-Quantize or Quantize-then-Prune? Understanding the Impact of Compression Order in Joint Model Compressioncited by 2
  7. 2024Resource-aware Mixed-precision Quantization for Enhancing Deployability of Transformers for Time-series Forecasting on Embedded FPGAscited by 1
  8. 2024BATON: Enhancing Batch-wise Inference Efficiency for Large Language Models via Dynamic Re-batchingcited by 1
  9. 2024Systolic Arrays and Structured Pruning Co-design for Efficient Transformers in Edge Systemscited by 1
  10. 2024A Comparative Study of Pruning Methods in Transformer-based Time Series Forecastingcited by 1
  11. 2025Private Transformer Inference in MLaaS: A Surveycited by 1
  12. 2025LinDeps: A Fine-tuning Free Post-Pruning Method to Remove Layer-Wise Linear Dependencies with Guaranteed Performance Preservationcited by 1
  13. 2025I-Segmenter: Integer-Only Vision Transformer for Efficient Semantic Segmentationcited by 1
  14. 2025Keeping it Local, Tiny and Real: Automated Report Generation on Edge Computing Devices for Mechatronic-Based Cognitive Systemscited by 1
  15. 2026Unsupervised Modular Adaptive Region Growing and RegionMix Classification for Wind Turbine Segmentationcited by 1
  16. 2025PHEONA: An Evaluation Framework for Large Language Model-based Approaches to Computational Phenotypingcited by 0
  17. 2025Dynamic Rank Reinforcement Learning for Adaptive Low-Rank Multi-Head Self Attention in Large Language Modelscited by 0
  18. 2026ViTMAlis: Towards Latency-Critical Mobile Video Analytics with Vision Transformerscited by 0
  19. 2026End-to-End Image Compression with Segmentation Guided Dual Coding for Wind Turbinescited by 0
  20. 2026Depth Adaptive Efficient Visual Autoregressive Modelingcited by 0
  21. 2026AI-IoT-Robotics Integration: Survey of Frameworks, Emerging Trends, and the Path Toward Connected Roboticscited by 0
  22. 2026Prefill/Decode-Aware Evaluation of LLM Inference on Emerging AI Acceleratorscited by 0
  23. 2026Recent Developments in Transformer Inference Deployment on FPGA Platforms: A Surveycited by 0