mlsanityData to 14 Oct 2026

Automated Tensor Scheduling for Hybrid CPU-GPU LLM Inference on Consumer Devices

arXiv 2607.10183 ↗First posted 11 Jul 2026cs.DC cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Running large language models on consumer devices such as laptops and desktops is challenging because model weights often exceed GPU memory capacity, making offloading inference necessary to extend effective model capacity with CPU memory. Existing offloading systems, however, typically rely on coarse layer-level or expert-level scheduling, which overlooks substantial heterogeneity among tensors within the same layer and adapts poorly to changing hardware load conditions on such devices. This paper presents ATSInfer, a hybrid CPU-GPU inference system for consumer devices that performs offloading at tensor granularity. ATSInfer combines static tensor placement with load-aware dynamic transfer, and introduces asynchronous CPU-GPU coordination to efficiently schedule hardware storage, data movement, and computation across heterogeneous backends. We implement ATSInfer and evaluate it on representative consumer platforms using both dense and MoE models. Compared with existing systems, ATSInfer improves prefill throughput by up to 1.94× and decode throughput by up to 3.29×, while also increasing GPU utilization and making more effective use of PCIe bandwidth. These results show that ATSInfer can substantially improve the user experience of local LLM deployment on personal consumer devices.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 23 of its 37 references that are in the map. First posted on arXiv 11 Jul 2026.

Builds on

23 of 37 references.

References · 23

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  7. 2020Knowledge Distillation: A Surveycited by 1,603
  8. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  9. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  10. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  11. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  12. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
11 more
  1. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  2. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  3. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  4. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 177
  5. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUscited by 73
  6. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Modelscited by 71
  7. 2024MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cachecited by 50
  8. 2024AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inferencecited by 42
  9. 2026The Llama 4 Herd: Architecture, Training, Evaluation, and Deployment Notescited by 30
  10. 2025HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inferencecited by 16
  11. 2026DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025SpecOffload: Unlocking Latent GPU Capacity for LLM Inference on Resource-Constrained Devices
  2. 2026OmniPilot: An Uncertainty-Aware LLM Inference Advisor for Heterogeneous GPU Clusters ↗
  3. 2026LLMVisor: A Real-Time Latency Attribution Model for Multi-Tenant LLM Serving
  4. 2026Scaling LLM Inference Beyond Amdahl`s Limits via Eliminating Non-Scalable Overheads ↗
  5. 2025APEX: Asynchronous Parallel CPU-GPU Execution for Online LLM Inference on Constrained GPUs
  6. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference
  7. 2025Challenging GPU Dominance: When CPUs Outperform for On-Device LLM Inference
  8. 2025Cronus: Efficient LLM inference on Heterogeneous GPU Clusters via Partially Disaggregated Prefill