# Agent.xpu: Efficient Scheduling of Agentic LLM Workloads on Heterogeneous SoC

arXiv: https://arxiv.org/abs/2506.24045 · HTML: https://mlsanity.com/paper/2506.24045/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 53 of its 101 references that are in the map.
- First posted on arXiv 30 Jun 2025.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 30 of 53)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [CogAgent: A Visual Language Model for GUI Agents](https://mlsanity.com/paper/2312.08914/) (2023)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024)
- [FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU](https://mlsanity.com/paper/2303.06865/) (2023)
- [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023)
- [Efficiently Scaling Transformer Inference](https://mlsanity.com/paper/2211.05102/) (2022)
- [SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization](https://mlsanity.com/paper/1911.12237/) (2019)
- [Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve](https://mlsanity.com/paper/2403.02310/) (2024)
- [LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset](https://mlsanity.com/paper/2309.11998/) (2023)
- [Understanding the planning of LLM agents: A survey](https://mlsanity.com/paper/2402.02716/) (2024)
- [AppAgent: Multimodal Agents as Smartphone Users](https://mlsanity.com/paper/2312.13771/) (2023)
- [Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security](https://mlsanity.com/paper/2401.05459/) (2024)
- [PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU](https://mlsanity.com/paper/2312.12456/) (2023)
- [AutoDroid: LLM-powered Task Automation in Android](https://mlsanity.com/paper/2308.15272/) (2023)

## Built on by (most cited first, 11 of 11)

- [What Is Your AI Agent Buying? Evaluation, Biases, Model Dependence, & Emerging Implications for Agentic E-Commerce](https://mlsanity.com/paper/2508.02630/) (2025)
- [Accelerating Mobile Language Model via Speculative Decoding and NPU-Coordinated Execution](https://mlsanity.com/paper/2510.15312/) (2025)
- [Adaptive GPU Resource Allocation for Multi-Agent Collaborative Reasoning in Serverless Environments](https://mlsanity.com/paper/2512.22149/) (2025)
- [FastTTS: Accelerating Test-Time Scaling for Edge LLM Reasoning](https://mlsanity.com/paper/2509.00195/) (2025)
- [GUI Agents with Reinforcement Learning: Toward Digital Inhabitants](https://mlsanity.com/paper/2604.27955/) (2026)
- [Efficient Mixture-of-Experts LLM Inference with Apple Silicon NPUs](https://mlsanity.com/paper/2604.18788/) (2026)
- [A Policy-Driven Runtime Layer for Agentic LLM Serving](https://mlsanity.com/paper/2605.27744/) (2026)
- [Agents in the Large: Perception-Centered Architecture for Persistent Agents](https://mlsanity.com/paper/2608.30478/) (2026)
- [AgentKernel: The Trust-Native Agentic Operating System](https://mlsanity.com/paper/2609.29647/) (2026)
- [Empowering Hybrid Attention Models on NPUs](https://mlsanity.com/paper/2609.32114/) (2026)
- [Foundations of Proactive Agents: Principles, Technical Layers, and Proactivity-Gym](https://mlsanity.com/paper/2609.37267/) (2026)

## Abstract (arXiv)

Personal LLM agents increasingly combine foreground reactive interactions with background proactive monitoring, forming long-lived, stateful LLM flows that interleave prefill and token-by-token decode. While modern heterogeneous SoCs integrate CPUs, iGPUs, and NPUs to support on-device intelligence, existing LLM engines assume static, single-shot inference and lack mechanisms for flow-level concurrency, prioritization, and efficient accelerator coordination. As a result, commodity SoCs remain poorly matched to the dynamic, mixed-criticality execution patterns of personal agents. This paper presents Agent.xpu, the first LLM engine that orchestrates concurrent reactive and proactive LLM flows on commodity SoCs. Extensive profiling uncovers unique SoC characteristics of operator-accelerator affinity, asymmetric DDR contention, and stage-divergent batching behaviors distinct from cloud-serving assumptions. Agent.xpu introduces three key techniques: a heterogeneous execution graph (HEG) capturing NPU/iGPU affinity and elastic operator binding; flow-aware NPU-iGPU coordination with stage elasticity, decoupling prefill and decode to reduce bandwidth contention and enforce priorities; and fine-grained preemption with slack-aware piggybacking to guarantee reactive responsiveness without starving proactive work. Across realistic personal-agent workloads, Agent.xpu delivers 1.2-4.9× proactive throughput and reduces reactive latency by at least 91%, compared with both industrial iGPU-only serving engine and NPU-iGPU static inference with optimal tensor-partitioning schemes. Agent.xpu also minimizes energy consumption and graphics interference via controlled iGPU usage.