mlsanityData to 14 Oct 2026

APEX: Asynchronous Parallel CPU-GPU Execution for Online LLM Inference on Constrained GPUs

arXiv 2506.03296 ↗First posted 3 Jun 2025cs.DCcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Deploying large language models (LLMs) for online inference is often constrained by limited GPU memory, particularly due to the growing KV cache during auto-regressive decoding. Hybrid GPU-CPU execution has emerged as a promising solution by offloading KV cache management and parts of attention computation to the CPU. However, a key bottleneck remains: existing schedulers fail to effectively overlap CPU-offloaded tasks with GPU execution during the latency-critical, bandwidth-bound decode phase. This particularly penalizes real-time, decode-heavy applications (e.g., chat, Chain-of-Thought reasoning) which are currently underserved by existing systems, especially under memory pressure typical of edge or low-cost deployments. We present APEX, a novel, profiling-informed scheduling strategy that maximizes CPU-GPU parallelism during hybrid LLM inference. Unlike systems relying on static rules or purely heuristic approaches, APEX dynamically dispatches compute across heterogeneous resources by predicting execution times of CPU and GPU subtasks to maximize overlap while avoiding scheduling overheads. We evaluate APEX on diverse workloads and GPU architectures (NVIDIA T4, A10), using LLaMa-2-7B and LLaMa-3.1-8B models. Compared to GPU-only schedulers like vLLM, APEX improves throughput by 84% - 96% on T4 and 11% - 89% on A10 GPUs, while preserving latency. Against the best existing hybrid schedulers, it delivers up to 72% (T4) and 37% (A10) higher throughput in long-output settings. APEX significantly advances hybrid LLM inference efficiency on such memory-constrained hardware and provides a blueprint for scheduling in heterogeneous AI systems, filling a critical gap for efficient real-time LLM applications.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 26 of its 42 references that are in the map. First posted on arXiv 3 Jun 2025.

Builds on

26 of 42 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference2 Nov
  2. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism15 Apr
  3. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines18 Mar
  4. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memory12 Dec

Other references · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  11. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  12. 2022Efficiently Scaling Transformer Inferencecited by 539
10 more
  1. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 262
  2. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  3. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 202
  4. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 163
  5. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 112
  6. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 103
  7. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUscited by 73
  8. 2024AQUA: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domainscited by 14
  9. 2024HeteGen: Heterogeneous Parallel Inference for Large Language Models on Resource-Constrained Devicescited by 9
  10. 2024PIM-AI: A Novel Architecture for High-Efficiency LLM Inferencecited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs
  2. 2025Efficient LLM Inference with Activation Checkpointing and Hybrid Caching
  3. 2025Parallax: Efficient LLM Inference Service over Decentralized Environment
  4. 2025Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scaling
  5. 2024Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inference
  6. 2024Inference Performance Optimization for Large Language Models on CPUs
  7. 2025SpecOffload: Unlocking Latent GPU Capacity for LLM Inference on Resource-Constrained Devices
  8. 2025LeMix: Unified Scheduling for LLM Training and Inference on Multi-GPU Systems