mlsanityData to 14 Oct 2026

PIPO: Pipelined Offloading for Efficient Inference on Consumer Devices

arXiv 2504.03664 ↗First posted 15 Mar 2025cs.DC cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

The high memory and computation demand of large language models (LLMs) makes them challenging to be deployed on consumer devices due to limited GPU memory. Offloading can mitigate the memory constraint but often suffers from low GPU utilization, leading to low inference efficiency. In this work, we propose a novel framework, called pipelined offloading (PIPO), for efficient inference on consumer devices. PIPO designs a fine-grained offloading pipeline, complemented with optimized data transfer and computation, to achieve high concurrency and efficient scheduling for inference. Experimental results show that compared with state-of-the-art baseline, PIPO increases GPU utilization from below 40% to over 90% and achieves up to 3.1× higher throughput, running on a laptop equipped with a RTX3060 GPU of 6GB memory.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 20 references that are in the map. First posted on arXiv 15 Mar 2025.

Builds on

18 of 20 references.

References · 18

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  4. 2024Mixtral of Expertscited by 1,823
  5. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  6. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  7. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  8. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  9. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  10. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
  11. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  12. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
6 more
  1. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  2. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 252
  3. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  4. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
  5. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 177
  6. 2024VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Modelscited by 57

Built on by

Not cited yet.