# AgBench: Agentic AI Benchmarks for Personal AI Devices

arXiv: https://arxiv.org/abs/2609.38652 · HTML: https://mlsanity.com/paper/2609.38652/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 18 of its 33 references that are in the map.
- First posted on arXiv 29 Sep 2026.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 18 of 18)

- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://mlsanity.com/paper/2601.11868/) (2026)
- [Better & Faster Large Language Models via Multi-token Prediction](https://mlsanity.com/paper/2404.19737/) (2024)
- [Agent S: An Open Agentic Framework that Uses Computers Like a Human](https://mlsanity.com/paper/2410.08164/) (2024)
- [The OpenHands Software Agent SDK: A Composable and Extensible Foundation for Production Agents](https://mlsanity.com/paper/2511.03690/) (2025)
- [Agentic Much? Adoption of Coding Agents on GitHub](https://mlsanity.com/paper/2601.18341/) (2026)
- [ReAcTree: Hierarchical LLM Agent Trees with Control Flow for Long-Horizon Task Planning](https://mlsanity.com/paper/2511.02424/) (2025)
- [TUA-Bench: A Benchmark for General-Purpose Terminal-Use Agents](https://mlsanity.com/paper/2606.28480/) (2026)
- [EcoAgent: An Efficient Device-Cloud Collaborative Multi-Agent Framework for Mobile Automation](https://mlsanity.com/paper/2505.05440/) (2025)
- [Procedural Knowledge Improves Agentic LLM Workflows](https://mlsanity.com/paper/2511.07568/) (2025)
- [MyPCBench: A Benchmark for Personally Intelligent Computer-Use Agents](https://mlsanity.com/paper/2606.16748/) (2026)
- [Mobile-Agent-RAG: Driving Smart Multi-Agent Coordination with Contextual Knowledge Empowerment for Long-Horizon Mobile Automation](https://mlsanity.com/paper/2511.12254/) (2025)
- [Agent-X: Full Pipeline Acceleration of On-device AI Agents](https://mlsanity.com/paper/2605.10380/) (2026)
- [Hera: Learning Long-Horizon Coordination for Device-Cloud Collaborative LLM Agents](https://mlsanity.com/paper/2605.24598/) (2026)
- [From LLM Inference to Agentic Workloads: Characterization and Implications for Serving Systems](https://mlsanity.com/paper/2608.15127/) (2026)
- [PalmClaw: A Native On-Device Agent Framework for Mobile Phones](https://mlsanity.com/paper/2607.13027/) (2026)
- [Benchmarking LLM Serving Systems for Agentic AI Workloads with XPerf](https://mlsanity.com/paper/2608.20370/) (2026)

## Abstract (arXiv)

Agentic AI systems increasingly rely on cloud-hosted large language models for planning, tool use, and iterative execution, raising concerns about API cost and data exposure. Advances in personal AI devices enable agents to execute locally, but limited resources on device may affect task success and performance. Existing benchmarks are inadequate for systematically characterizing these trade-offs across devices, workloads, and deployment architectures. We present AgBench, a benchmark suite and open artifacts for reproducible evaluation of agentic AI on personal devices. Using AgBench, we evaluate local, hybrid, and cloud execution across agentic workloads, examining task success, latency, cloud API cost, and data exposure. Our results, drawn from over 162.07 million data points, show that personal AI devices can complete many agent tasks locally, but local-only execution generally has lower task success and longer completion times than cloud-only execution, especially as concurrency increases. Local-only execution eliminates cloud model API costs and sensitive-information exposure to cloud agents. Hybrid execution can improve task success, but its cloud cost and data exposure depend on how agents divide work and share information. No single architecture performs best across task success, goodput, cloud cost, and data exposure; deployment choices should reflect the intended workload and device capabilities. AgBench is available at https://anonymous.4open.science/r/AgBench-2777.