mlsanityData to 7 Oct 2026

LLM Inference Unveiled: Survey and Roofline Model Insights

arXiv 2402.16363 ↗First posted 26 Feb 2024cs.CL cs.AIcited by 147

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

The field of efficient Large Language Model (LLM) inference is rapidly evolving, presenting a unique blend of opportunities and challenges. Although the field has expanded and is vibrant, there hasn't been a concise framework that analyzes the various methods of LLM Inference to provide a clear understanding of this domain. Our survey stands out from traditional literature reviews by not only summarizing the current state of research but also by introducing a framework based on roofline model for systematic analysis of LLM inference techniques. This framework identifies the bottlenecks when deploying LLMs on hardware devices and provides a clear understanding of practical problems, such as why LLMs are memory-bound, how much memory and computation they need, and how to choose the right hardware. We systematically collate the latest advancements in efficient LLM inference, covering crucial areas such as model compression (e.g., Knowledge Distillation and Quantization), algorithm improvements (e.g., Early Exit and Mixture-of-Expert), and both hardware and system-level enhancements. Our survey stands out by analyzing these methods with roofline model, helping us understand their impact on memory access and computation. This distinctive approach not only showcases the current research landscape but also delivers valuable insights for practical implementation, positioning our work as an indispensable resource for researchers new to the field as well as for those seeking to deepen their understanding of efficient LLM deployment. The analyze tool, LLM-Viewer, is open-sourced.

Key facts. Cited by 147 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 182 of its 234 references that are in the map. First posted on arXiv 26 Feb 2024.

Builds on

182 of 234 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAs8 Jan
  2. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems23 Dec
  3. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memory12 Dec
  4. 2023FlashDecoding++: Faster Large Language Model Inference on GPUs2 Nov
  5. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills31 Aug

Other references · 177

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2020Longformer: The Long-Document Transformercited by 3,362
  12. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
38 more
  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  2. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  3. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  5. 2023A Survey of Large Language Modelscited by 2,537
  6. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  10. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  11. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  12. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  13. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  14. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  15. 2020Knowledge Distillation: A Surveycited by 1,578
  16. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  17. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  18. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  19. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  20. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  21. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  22. 2023A Survey on Multimodal Large Language Modelscited by 898
  23. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  24. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  25. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  26. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  27. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  28. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  29. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  30. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  31. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  32. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  33. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  34. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  35. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  36. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  37. 2017Non-Autoregressive Neural Machine Translationcited by 457
  38. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 456

50 most cited of 177 All 177 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 42; 2025: 63; 2026: 42.

147 papers.

Sort:

By area:

  1. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  2. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 394
  3. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  4. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Tokencited by 164
  5. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  6. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 123
  7. 2024Matryoshka Multimodal Modelscited by 82
  8. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  9. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  10. 2024AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruningcited by 43
  11. 2024SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Modelscited by 41
  12. 2024Filter, Correlate, Compress: Training-Free Token Reduction for MLLM Accelerationcited by 38
38 more
  1. 2025Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantizationcited by 36
  2. 2025MQuant: Unleashing the Inference Potential of Multimodal Large Language Models via Full Static Quantizationcited by 35
  3. 2025Laminar: A Scalable Asynchronous RL Post-Training Frameworkcited by 35
  4. 2025TiDAR: Think in Diffusion, Talk in Autoregressioncited by 35
  5. 2025Global Compression Commander: Plug-and-Play Inference Acceleration for High-Resolution Large Vision-Language Modelscited by 32
  6. 2025Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inferencecited by 32
  7. 2024ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterizationcited by 27
  8. 2025Parallel Loop Transformer for Efficient Test-Time Computation Scalingcited by 23
  9. 2024SpeechPrune: Context-aware Token Pruning for Speech Information Retrievalcited by 20
  10. 2024InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inferencecited by 19
  11. 2024LLMCO2: Advancing Accurate Carbon Footprint Prediction for LLM Inferencescited by 19
  12. 2024Transformer-Lite: High-efficiency Deployment of Large Language Models on Mobile Phone GPUscited by 17
  13. 2024Progressive Mixed-Precision Decoding for Efficient LLM Inferencecited by 17
  14. 2025MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoEcited by 17
  15. 2026How Fast Can I Run My VLA? Demystifying VLA Inference Performance with VLA-Perfcited by 17
  16. 2024Intent3D: 3D Object Detection in RGB-D Scans Based on Human Intentioncited by 16
  17. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  18. 2025INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formatscited by 16
  19. 2025Seesaw: High-throughput LLM Inference via Model Re-shardingcited by 15
  20. 2025Scaling Law for Quantization-Aware Trainingcited by 15
  21. 2025The Pitfalls of KV Cache Compressioncited by 15
  22. 2024Model Quantization and Hardware Acceleration for Vision Transformers: A Comprehensive Surveycited by 13
  23. 2024Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM modelscited by 12
  24. 2025Benchmarking Post-Training Quantization in LLMs: Comprehensive Taxonomy, Unified Evaluation, and Comparative Analysiscited by 12
  25. 2025Learning to Inference Adaptively for Multimodal Large Language Modelscited by 12
  26. 2024PrefixQuant: Eliminating Outliers by Prefixed Tokens for Large Language Models Quantizationcited by 11
  27. 2025SplitReason: Learning To Offload Reasoningcited by 11
  28. 2025Kinetics: Rethinking Test-Time Scaling Lawscited by 11
  29. 2024Interpolating Video-LLMs: Toward Longer-sequence LMMs in a Training-free Mannercited by 10
  30. 2025Cognitive Edge Computing: A Comprehensive Survey on Optimizing Large Models and AI Agents for Pervasive Deploymentcited by 10
  31. 2024Performance Modeling and Workload Analysis of Distributed Large Language Model Training and Inferencecited by 9
  32. 2025LP-Spec: Leveraging LPDDR PIM for Efficient LLM Mobile Speculative Inference with Architecture-Dataflow Co-Optimizationcited by 9
  33. 2024Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Formatcited by 8
  34. 2025BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Efficient LLM Inferencecited by 8
  35. 2025Characterizing and Optimizing LLM Inference Workloads on CPU-GPU Coupled Architecturescited by 8
  36. 2025ButterflyQuant: Ultra-low-bit LLM Quantization through Learnable Orthogonal Butterfly Transformscited by 8
  37. 2025MixKVQ: Query-Aware Mixed-Precision KV Cache Quantization for Long-Context Reasoningcited by 8
  38. 2024LLM-dCache: Improving Tool-Augmented LLMs with GPT-Driven Localized Data Cachingcited by 7

50 most cited of 147 All 147 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Efficient LLM Inference on CPUs
  2. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline
  3. 2024Faster and Lighter LLMs: A Survey on Current Challenges and Way Forward
  4. 2025Optimizing LLMs for Resource-Constrained Environments: A Survey of Model Compression Techniques
  5. 2024LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scale
  6. 2025Reimagining Memory Access for LLM Inference: Compression-Aware Memory Controller Design
  7. 2025Can abstract concepts from LLM improve SLM performance?
  8. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective