mlsanityData to 14 Oct 2026

FlexInfer: Breaking Memory Constraint via Flexible and Efficient Offloading for On-Device LLM Inference

arXiv 2503.03777 ↗First posted 4 Mar 2025cs.OS cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) face challenges for on-device inference due to high memory demands. Traditional methods to reduce memory usage often compromise performance and lack adaptability. We propose FlexInfer, an optimized offloading framework for on-device inference, addressing these issues with techniques like asynchronous prefetching, balanced memory locking, and flexible tensor preservation. These strategies enhance memory efficiency and mitigate I/O bottlenecks, ensuring high performance within user-specified resource constraints. Experiments demonstrate that FlexInfer significantly improves throughput under limited resources, achieving up to 12.5 times better performance than existing methods and facilitating the deployment of large models on resource-constrained devices.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 26 of its 51 references that are in the map. First posted on arXiv 4 Mar 2025.

Builds on

26 of 51 references.

References · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  7. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  8. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  9. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  10. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  11. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  12. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
14 more
  1. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 252
  2. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  3. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 177
  4. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 103
  5. 2019Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerializationcited by 91
  6. 2024OneBit: Towards Extremely Low-bit Large Language Modelscited by 79
  7. 2023LLM-Rec: Personalized Recommendation via Prompting Large Language Modelscited by 77
  8. 2024An empirical study of LLaMA3 quantization: from LLMs to MLLMscited by 64
  9. 2023LLMCad: Fast and Scalable On-device Large Language Model Inferencecited by 45
  10. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  11. 2022POET: Training Neural Networks on Tiny Devices with Integrated Rematerialization and Pagingcited by 26
  12. 2024CHESS: Optimizing LLM Inference via Channel-Wise Thresholding and Selective Sparsificationcited by 7
  13. 2025When Compression Meets Model Compression: Memory-Efficient Double Compression for Large Language Modelscited by 3
  14. 2025EvoP: Robust LLM Inference via Evolutionary Pruningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025LayerScope: Predictive Cross-Layer Scheduling for Efficient Multi-Batch MoE Inference on Legacy Serverscited by 8
  2. 2025RAGDoll: Efficient Offloading-based Online RAG System on a Single GPUcited by 2
  3. 2025Characterizing State Space Model and Hybrid Language Model Performance with Long Contextcited by 1
  4. 2026JustFit: Just-in-Time State Management for Local LLM Servingcited by 0