mlsanityData to 14 Oct 2026

LLM in a flash: Efficient Large Language Model Inference with Limited Memory

arXiv 2312.11514 ↗First posted 12 Dec 2023cs.CL cs.AI cs.LGcited by 177

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large language models (LLMs) are central to modern natural language processing, delivering exceptional performance in various tasks. However, their substantial computational and memory requirements present challenges, especially for devices with limited DRAM capacity. This paper tackles the challenge of efficiently running LLMs that exceed the available DRAM capacity by storing the model parameters in flash memory, but bringing them on demand to DRAM. Our method involves constructing an inference cost model that takes into account the characteristics of flash memory, guiding us to optimize in two critical areas: reducing the volume of data transferred from flash and reading data in larger, more contiguous chunks. Within this hardware-informed framework, we introduce two principal techniques. First, "windowing" strategically reduces data transfer by reusing previously activated neurons, and second, "row-column bundling", tailored to the sequential data access strengths of flash memory, increases the size of data chunks read from flash memory. These methods collectively enable running models up to twice the size of the available DRAM, with a 4-5x and 20-25x increase in inference speed compared to naive loading approaches in CPU and GPU, respectively. Our integration of sparsity awareness, context-adaptive loading, and a hardware-oriented design paves the way for effective inference of LLMs on devices with limited memory.

Key facts. Cited by 177 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 28 of its 55 references that are in the map. First posted on arXiv 12 Dec 2023.

Builds on

28 of 55 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  8. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  9. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  10. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  11. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  12. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 506
16 more
  1. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  2. 2016EIE: Efficient Inference Engine on Compressed Deep Neural Networkcited by 295
  3. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  4. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
  5. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 252
  6. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
  7. 2020Sparse GPU Kernels for Deep Learningcited by 141
  8. 2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 87
  9. 2023Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decodingcited by 80
  10. 2023Intriguing Properties of Quantization at Scalecited by 49
  11. 2018Neural Cache: Bit-Serial In-Cache Acceleration of Deep Neural Networkscited by 32
  12. 2023Compress, Then Prompt: Improving Accuracy-Efficiency Trade-off of LLM Inference with Transferable Promptcited by 28
  13. 2023Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsitycited by 17
  14. 2023A Hardware Evaluation Framework for Large Language Model Inferencecited by 14
  15. 2023Alternating Updates for Efficient Transformerscited by 13
  16. 2023Contextual Distortion Reveals Constituency: Masked Language Models are Implicit Parserscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 52; 2025: 57; 2026: 67.

177 papers.

Sort:

By area:

  1. 2025AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challengescited by 169
  2. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 163
  3. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  4. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 146
  5. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 112
  6. 2024Small Language Models: Survey, Measurements, and Insightscited by 112
  7. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 102
  8. 2024Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunitiescited by 83
  9. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 78
  10. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Modelscited by 76
  11. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Modelscited by 71
  12. 2024ReLU2 Wins: Discovering Efficient Activation Functions for Sparse LLMscited by 67
38 more
  1. 2024Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inferencecited by 61
  2. 2024Training-Free Activation Sparsity in Large Language Modelscited by 55
  3. 2025KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environmentscited by 51
  4. 2024Harnessing the Power of Multiple Minds: Lessons Learned from LLM Routingcited by 46
  5. 2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devicescited by 46
  6. 2024MELTing point: Mobile Evaluation of Language Transformerscited by 36
  7. 2024Block Transformer: Global-to-Local Language Modeling for Fast Inferencecited by 32
  8. 2024Exploring Autonomous Agents through the Lens of Large Language Models: A Reviewcited by 31
  9. 2025TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platformscited by 29
  10. 2024Wings: Learning Multimodal LLMs without Text-only Forgettingcited by 23
  11. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 23
  12. 2024Configurable Foundation Models: Building LLMs from a Modular Perspectivecited by 22
  13. 2025LLM-Powered AI Agent Systems and Their Applications in Industrycited by 22
  14. 2025R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inferencecited by 21
  15. 2024InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inferencecited by 19
  16. 2024Salute the Classic: Revisiting Challenges of Machine Translation in the Age of Large Language Modelscited by 17
  17. 2024A Survey on Hardware Accelerators for Large Language Modelscited by 16
  18. 2024Efficient Non-Parametric Uncertainty Quantification for Black-Box Large Language Models and Decision Planningcited by 16
  19. 2024CoreInfer: Accelerating Large Language Model Inference with Semantics-Inspired Adaptive Sparse Activationcited by 14
  20. 2024Mixture of Cache-Conditional Experts for Efficient Mobile Device Inferencecited by 14
  21. 2024Accelerating Retrieval-Augmented Generationcited by 14
  22. 2025Pretraining with hierarchical memories: separating long-tail and common knowledgecited by 14
  23. 2025RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inferencecited by 13
  24. 2025Agentic Learner with Grow-and-Refine Multimodal Semantic Memorycited by 13
  25. 2026AgentCPM-Explore: Realizing Long-Horizon Deep Exploration for Edge-Scale Agentscited by 12
  26. 2024Analyzing and Adapting Large Language Models for Few-Shot Multilingual NLU: Are We There Yet?cited by 11
  27. 2024KV Prediction for Improved Time to First Tokencited by 11
  28. 2024Efficient LLM Inference using Dynamic Input Pruning and Cache-Aware Maskingcited by 11
  29. 2025BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Efficient LLM Inferencecited by 11
  30. 2024Entity Retrieval for Answering Entity-Centric Questionscited by 10
  31. 2025TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Schedulingcited by 10
  32. 2024Chameleon: Adaptive Caching and Scheduling for Many-Adapter LLM Inference Environmentscited by 9
  33. 2025FloE: On-the-Fly MoE Inference on Memory-constrained GPUcited by 9
  34. 2025SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsitycited by 9
  35. 2026Forget to Improve: On-Device LLM-Agent Continual Learning via Budget-Curated Memorycited by 9
  36. 2025CoreMatching: A Co-adaptive Sparse Inference Framework with Token and Neuron Pruning for Comprehensive Acceleration of Vision-Language Modelscited by 8
  37. 2025MNN-LLM: A Generic Inference Engine for Fast Large Language Model Deployment on Mobile Devicescited by 8
  38. 2024HiRE: High Recall Approximate Top-k Estimation for Efficient LLM Inferencecited by 7

50 most cited of 177 All 177 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Understand and Accelerate Memory Processing Pipeline for Large Language Model Inference
  2. 2023FlashDecoding++: Faster Large Language Model Inference on GPUs
  3. 2025Make LLM Inference Affordable to Everyone: Augmenting GPU Memory with NDP-DIMM
  4. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving
  5. 2024Flash Communication: Reducing Tensor Parallelization Bottleneck for Fast Large Language Model Inference
  6. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective
  7. 2025PIM Is All You Need: A CXL-Enabled GPU-Free System for Large Language Model Inference
  8. 2024A Survey on Large Language Model Acceleration based on KV Cache Management