mlsanityData to 7 Oct 2026

FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU

arXiv 2303.06865 ↗First posted 13 Mar 2023cs.LG cs.AI cs.PFcited by 567

Not on the map. Why?

Cited by 305 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The high computational and memory requirements of large language model (LLM) inference make it feasible only with multiple high-end accelerators. Motivated by the emerging demand for latency-insensitive tasks with batched processing, this paper initiates the study of high-throughput LLM inference using limited resources, such as a single commodity GPU. We present FlexGen, a high-throughput generation engine for running LLMs with limited GPU memory. FlexGen can be flexibly configured under various hardware resource constraints by aggregating memory and computation from the GPU, CPU, and disk. By solving a linear programming problem, it searches for efficient patterns to store and access tensors. FlexGen further compresses the weights and the attention cache to 4 bits with negligible accuracy loss. These techniques enable FlexGen to have a larger space of batch size choices and thus significantly increase maximum throughput. As a result, when running OPT-175B on a single 16GB GPU, FlexGen achieves significantly higher throughput compared to state-of-the-art offloading systems, reaching a generation throughput of 1 token/s for the first time with an effective batch size of 144. On the HELM benchmark, FlexGen can benchmark a 30B model with a 16GB GPU on 7 representative sub-scenarios in 21 hours. The code is available at https://github.com/FMInference/FlexGen

Key facts. Cited by 567 papers in the map (data through 7 Oct 2026). Builds on 33 of its 47 references that are in the map. First posted on arXiv 13 Mar 2023.

Builds on

33 of 47 references.

References · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2016Pointer Sentinel Mixture Modelscited by 2,992
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  9. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  10. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  11. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  12. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
21 more
  1. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  2. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  3. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  4. 2022Efficiently Scaling Transformer Inferencecited by 529
  5. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  6. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  7. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  8. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  9. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  10. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
  11. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
  12. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  13. 2022Can Foundation Models Wrangle Your Data?cited by 173
  14. 2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 84
  15. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71
  16. 2023SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficientcited by 58
  17. 2018SuperNeurons: Dynamic GPU Memory Management for Training Deep Neural Networkscited by 55
  18. 2021SpreadsheetCoder: Formula Prediction from Semi-structured Contextcited by 41
  19. 2022AlphaTuning: Quantization-Aware Parameter-Efficient Adaptation of Large-Scale Pre-Trained Language Modelscited by 22
  20. 2022Harmony: Overcoming the Hurdles of GPU Memory Capacity to Train Massive DNN Models on Commodity Serverscited by 6
  21. 2022OLLA: Optimizing the Lifetime and Location of Arrays to Reduce the Memory Usage of Neural Networkscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 53; 2024: 166; 2025: 151; 2026: 197.

567 papers.

Sort:

By area:

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  3. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  4. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  5. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  6. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  7. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  8. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  9. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  10. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  11. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  12. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
38 more
  1. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  2. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  3. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 359
  4. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
  5. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  6. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
  7. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  8. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 253
  9. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  10. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 216
  11. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 216
  12. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  13. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  14. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  15. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  16. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  17. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187
  18. 2023Efficient Large Language Models: A Surveycited by 183
  19. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 170
  20. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  21. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
  22. 2023Accelerating LLM Inference with Staged Speculative Decodingcited by 151
  23. 2023REST: Retrieval-Based Speculative Decodingcited by 148
  24. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inferencecited by 148
  25. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  26. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 146
  27. 2023Fast Distributed Inference Serving for Large Language Modelscited by 143
  28. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  29. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  30. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 132
  31. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  32. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 124
  33. 2023Accelerating Transformer Inference for Translation via Parallel Decodingcited by 123
  34. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attentioncited by 122
  35. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 122
  36. 2023SparQ Attention: Bandwidth-Efficient LLM Inferencecited by 119
  37. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  38. 2024Language Model Cascades: Token-level uncertainty and beyondcited by 115

50 most cited of 567 All 567 (JSON)