mlsanityData to 7 Oct 2026

Agent-X: Full Pipeline Acceleration of On-device AI Agents

arXiv 2605.10380 ↗First posted 11 May 2026cs.AIcited by 4

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

LLM-based agents deliver state-of-the-art performance across tasks but incur high end-to-end latency on edge devices. We introduce Agent-X, a software-only, accuracy-preserving framework that accelerates both the prefill and decode stages of on-device agent workloads. Agent-X's two key components rewrite prompts to leverage prefix caching tailored to agent-specific input-token patterns and enable LLM-free speculative decoding for fast token generation with minimal overhead. On representative agentic workloads, Agent-X achieves a 1.61x end-to-end speedup in real systems with no accuracy loss and can be seamlessly integrated into existing on-device AI agents. To the best of our knowledge, ours is the first to systematically characterize and eliminate latency bottlenecks in on-device agents.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 35 of its 63 references that are in the map. First posted on arXiv 11 May 2026.

Builds on

35 of 63 references.

References · 35

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  6. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  7. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  8. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  9. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  10. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  11. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  12. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 551
23 more
  1. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  2. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
  3. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 359
  4. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 283
  5. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  6. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 211
  7. 2025Plan-and-Act: Improving Planning of Agents for Long-Horizon Taskscited by 190
  8. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187
  9. 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaborationcited by 187
  10. 2023An LLM Compiler for Parallel Function Callingcited by 170
  11. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 170
  12. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devicescited by 65
  13. 2024TinyAgent: Function Calling at the Edgecited by 55
  14. 2025PIM Is All You Need: A CXL-Enabled GPU-Free System for Large Language Model Inferencecited by 22
  15. 2025Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabulariescited by 20
  16. 2024EdgeLLM: A Highly Efficient CPU-FPGA Heterogeneous Edge Accelerator for Large Language Modelscited by 19
  17. 2024Cambricon-LLM: A Chiplet-Based Hybrid Architecture for On-Device Inference of 70B LLMcited by 17
  18. 2025CLONE: Customizing LLMs for Efficient Latency-Aware Inference at the Edgecited by 16
  19. 2024An LLM-Tool Compiler for Fused Parallel Function Callingcited by 14
  20. 2025EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devicescited by 10
  21. 2024DecDEC: A Systems Approach to Advancing Low-Bit LLM Quantizationcited by 7
  22. 2025Never Start from Scratch: Expediting On-Device LLM Personalization via Explainable Model Selectioncited by 7
  23. 2024FDC: Fast KV Dimensionality Compression for Efficient LLM Inferencecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026AgentDecarbonizer: Carbon-Aware Execution for AI Agentscited by 0
  2. 2026PastForward: Faster On-Device GUI Agents via Computational Experience Reusecited by 0
  3. 2026PReCache: Efficient KV Cache Sharing for Multi-LoRA Agents via Low-Rank Precomputation and Neutral Reconstructioncited by 0
  4. 2026AgBench: Agentic AI Benchmarks for Personal AI Devicescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026AgentStop: Terminating Local AI Agents Early to Save Energy in Consumer Devices
  2. 2026AgentPerfBench: A Benchmarking and Evaluation Suite for Inference Performance of Agentic LLMs
  3. 2025Efficient On-Device Agents via Adaptive Context Management
  4. 2026Heterogeneous Computing: The Key to Powering the Future of AI Agent Inference
  5. 2025FastTTS: Accelerating Test-Time Scaling for Edge LLM Reasoning
  6. 2025Speculative Actions: A Lossless Framework for Faster Agentic Systems
  7. 2026AgentSpec: Speculative Decoding for Batch Inference of LLM Agents
  8. 2026SwarmX: Agentic Scheduling for Low-Latency Agentic Systems