mlsanityData to 7 Oct 2026

Inference Optimization of Foundation Models on AI Accelerators

arXiv 2407.09111 ↗First posted 12 Jul 2024cs.AI cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Powerful foundation models, including large language models (LLMs), with Transformer architectures have ushered in a new era of Generative AI across various industries. Industry and research community have witnessed a large number of new applications, based on those foundation models. Such applications include question and answer, customer services, image and video generation, and code completions, among others. However, as the number of model parameters reaches to hundreds of billions, their deployment incurs prohibitive inference costs and high latency in real-world scenarios. As a result, the demand for cost-effective and fast inference using AI accelerators is ever more higher. To this end, our tutorial offers a comprehensive discussion on complementary inference optimization techniques using AI accelerators. Beginning with an overview of basic Transformer architectures and deep learning system frameworks, we deep dive into system optimization techniques for fast and memory-efficient attention computations and discuss how they can be implemented efficiently on AI accelerators. Next, we describe architectural elements that are key for fast transformer inference. Finally, we examine various model compression and fast decoding strategies in the same context.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 72 of its 116 references that are in the map. First posted on arXiv 12 Jul 2024.

Builds on

72 of 116 references.

References · 72

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  10. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  11. 2020Longformer: The Long-Document Transformercited by 3,362
  12. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
38 more
  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  3. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  5. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  6. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  7. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  8. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  9. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  10. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  11. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  12. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  13. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  14. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  15. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
  16. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  17. 2022On Distillation of Guided Diffusion Modelscited by 721
  18. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  19. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  20. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 610
  21. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  22. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  23. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  24. 2023The Falcon Series of Open Language Modelscited by 526
  25. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  26. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  27. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  28. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  29. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  30. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  31. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  32. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  33. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
  34. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 295
  35. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  36. 2022Post-training Quantization on Diffusion Modelscited by 262
  37. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 223
  38. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 198

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 3.

9 papers.

  1. 2024LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Acceleratorscited by 31
  2. 2026When LLMs get significantly worse: A statistical approach to detect model degradationscited by 5
  3. 2025A Proximal Operator for Inducing 2:4-Sparsitycited by 4
  4. 2025Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMscited by 4
  5. 2025Bench360: Benchmarking Local LLM Inference from 360 Degreescited by 4
  6. 2026Multi-Modal Data-Enhanced Foundation Models for Prediction and Control in Wireless Networks: A Surveycited by 3
  7. 2025Reducing Storage of Pretrained Neural Networks by Rate-Constrained Quantization and Entropy Codingcited by 0
  8. 2025SAIL: SRAM-Accelerated LLM Inference System with Lookup-Table-based GEMVcited by 0
  9. 2026Attention Quantization for Tabular Foundation Modelscited by 0