mlsanityData to 14 Oct 2026

Efficient On-Device Agents via Adaptive Context Management

arXiv 2511.03728 ↗First posted 24 Sep 2025cs.AI cs.LGcited by 1

On the map: Efficient LLM inference and serving → Efficient reasoning and agent inference

Abstract (arXiv)

On-device AI agents offer the potential for personalized, low-latency assistance, but their deployment is fundamentally constrained by limited memory capacity. Context in agentic settings worsens this problem due to large static tool schemas and a growing interaction history that continually expands the persistent KV cache. To maintain on-device feasibility, agents must operate near the minimum task-sufficient context, while preserving task performance. We introduce two complementary mechanisms: (1) a learned intra-session memory architecture that distills trajectories into an append-only Context State Object (CSO), preserving current state and relevant details from previous steps that may have future utility, while still supporting KV-cache reuse, and (2) a just-in-time schema-passing mechanism that loads full tool definitions only upon tool selection. We instantiate this framework by adapting 3B-parameter SLMs and profiling them on smartphone hardware. In our evaluations, CSO-based memory outperforms KV-cache compression and trained summarization while retaining performance near or even above full-history execution. Deployed on-device, our framework reduces initial tool context by over 6× and interaction-context growth by 10-25 times, substantially reducing initial

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Efficient reasoning and agent inference, one of 12 directions in Efficient LLM inference and serving. Builds on 25 of its 37 references that are in the map. First posted on arXiv 24 Sep 2025.

Builds on

25 of 37 references.

References · 25

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  4. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  5. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  6. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 1,002
  7. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 393
  8. 2025MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agentscited by 238
  9. 2025APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplaycited by 167
  10. 2025XAttention: Block Sparse Attention with Antidiagonal Scoringcited by 137
  11. 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Taskscited by 118
  12. 2024xLAM: A Family of Large Action Models to Empower AI Agent Systemscited by 111
13 more
  1. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoningcited by 97
  2. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstructioncited by 95
  3. 2022Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generationcited by 94
  4. 2023Recursively Summarizing Enables Long-Term Dialogue Memory in Large Language Modelscited by 62
  5. 2024TinyAgent: Function Calling at the Edgecited by 58
  6. 2024Hammer: Robust Function-Calling for On-Device Language Models via Function Maskingcited by 55
  7. 2024Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Taskscited by 52
  8. 2025KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environmentscited by 51
  9. 2024HAICOSYSTEM: An Ecosystem for Sandboxing Safety Risks in Human-AI Interactionscited by 44
  10. 2024Octopus: On-device language model for function calling of software APIscited by 27
  11. 2025Collaborating with AI Agents: Field Experiments on Teamwork, Productivity, and Performancecited by 14
  12. 2025Arch-Router: Aligning LLM Routing with Human Preferencescited by 9
  13. 2025On-Device LLM for Context-Aware Wi-Fi Roamingcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Actionable Hallucination Detection: Translating Latent Uncertainty into Agentic Critiquecited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026MAGNET: Towards Adaptive GUI Agents with Memory-Driven Knowledge Evolution
  2. 2026Agent-X: Full Pipeline Acceleration of On-device AI Agents
  3. 2026Just-In-Time Agent Memory with Runtime Agentic Research
  4. 2026LLM Agents Are Latent Context Managers: Eliciting Self-Managed Context via State Proprioception
  5. 2026PalmClaw: A Native On-Device Agent Framework for Mobile Phones
  6. 2026Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devices
  7. 2025Beyond Training: Enabling Self-Evolution of Agents with MOBIMEM
  8. 2026Addressable Recall Compaction for Long Context-Window Control in AI Agents