mlsanityData to 14 Oct 2026

SchemaWeaver: Accelerating On-Device Mobile Agentic AI via Schema-Guided Parallel LLM Decoding

arXiv 2610.17436 ↗First posted 12 Oct 2026cs.AIcited by 0

On the map: LLM agents and tool use → Agentic AI for wireless networks also fits: General LLM agent frameworks and benchmarks

also in 1 more
Abstract (arXiv)

Large language models (LLMs) on mobile devices can act as agents that call external software tools to complete complex tasks, but the major challenge of supporting on-device mobile agentic AI with LLMs is the long latency of LLM tool calling on mobile devices. In particular, the decode phase is the main contributor of such long latency, but is hard to speed up on mobile processors as it is bound by memory bandwidth rather than compute power. To address this limitation, in this paper we present SchemaWeaver, a new on-device mobile agentic AI technique that enforces schema-guided parallel decoding in the decode phase of LLM tool calling, to reduce its latency by producing multiple tokens in each decode pass. SchemaWeaver can be generically applied to the LLM inference engine on mobile devices without involving any training efforts or changes on LLMs, and our experiment results on major mobile SoCs and LLM families show that SchemaWeaver can reduce the LLM tool calling latency by up to 3.9 times, compared to the competitive baselines.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Agentic AI for wireless networks, one of 16 directions in LLM agents and tool use. Builds on 57 of its 77 references that are in the map. First posted on arXiv 12 Oct 2026.

Builds on

57 of 77 references.

References · 57

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  7. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  9. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  10. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  11. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  12. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
38 more
  1. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 578
  2. 2024Executable Code Actions Elicit Better LLM Agentscited by 564
  3. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 415
  4. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
  5. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 331
  6. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 298
  7. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 293
  8. 2025Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directionscited by 265
  9. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  10. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  11. 2023Grammar-Constrained Decoding for Structured NLP Tasks without Finetuningcited by 197
  12. 2023AutoDroid: LLM-powered Task Automation in Androidcited by 184
  13. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 177
  14. 2026DFlash: Block Diffusion for Flash Speculative Decodingcited by 135
  15. 2023Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generationcited by 105
  16. 2024On-Device Language Models: A Comprehensive Reviewcited by 104
  17. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Accelerationcited by 72
  18. 2024Fast On-device LLM Inference with NPUscited by 62
  19. 2024TinyAgent: Function Calling at the Edgecited by 58
  20. 2024Octopus v2: On-device language model for super agentcited by 47
  21. 2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Servicescited by 45
  22. 2024SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applicationscited by 36
  23. 2024AutoDroid-V2: Boosting SLM-based GUI Agents via Code Generationcited by 34
  24. 2024MobileFlow: A Multimodal LLM For Mobile GUI Agentcited by 32
  25. 2025Learning to Keep a Promise: Scaling Language Model Decoding Parallelism with Learned Asynchronous Decodingcited by 28
  26. 2024APAR: LLMs Can Do Auto-Parallel Auto-Regressive Decodingcited by 24
  27. 2025Characterizing Mobile SoC for Accelerating Heterogeneous LLM Inferencecited by 22
  28. 2026Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Servingcited by 20
  29. 2025MNN-LLM: A Generic Inference Engine for Fast Large Language Model Deployment on Mobile Devicescited by 8
  30. 2026XGrammar-2: Dynamic and Efficient Structured Generation Engine for Agentic LLMscited by 7
  31. 2025TinyLLM: Evaluation and Optimization of Small Language Models for Agentic Tasks on Edge Devicescited by 6
  32. 2026SPORK: Self-Speculative Forking to Accelerate Agentic LLM Inferencecited by 6
  33. 2024DroidCall: A Dataset for LLM-powered Android Intent Invocationcited by 5
  34. 2025Accelerating Mobile Language Model via Speculative Decoding and NPU-Coordinated Executioncited by 5
  35. 2026ToolSpec: Accelerating Tool Calling via Schema-Aware and Retrieval-Augmented Speculative Decodingcited by 5
  36. 2026Agent-X: Full Pipeline Acceleration of On-device AI Agentscited by 5
  37. 2025Streaming, Fast and Slow: Cognitive Load-Aware Streaming for Efficient LLM Servingcited by 3
  38. 2025MNN-AECS: Energy Optimization for LLM Decoding on Mobile Devices via Adaptive Core Selectioncited by 3

50 most cited of 57 All 57 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026AgentPerfBench: A Benchmarking and Evaluation Suite for Inference Performance of Agentic LLMs
  2. 2023Revolutionizing Mobile Interaction: Enabling a 3 Billion Parameter GPT LLM on Mobile
  3. 2024MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents
  4. 2026EdgeFlow: Fast Cold Starts for LLMs on Mobile Devices ↗
  5. 2025Combating the Memory Walls: Optimization Pathways for Long-Context Agentic LLM Inference
  6. 2023LinguaLinked: A Distributed Large Language Model Inference System for Mobile Devices
  7. 2025FastTTS: Accelerating Test-Time Scaling for Edge LLM Reasoning
  8. 2026End-to-End Latency-Minimizing and Load-Balanced Request Scheduling for Edge LLM Inference in Agentic AI Services