mlsanityData to 14 Oct 2026

BudgetBench: A Budget-Tiered Protocol and Pilot Harness for Memory Strategy Evaluation in Local Large Language Model Agents

arXiv 2609.13149 ↗First posted 1 Jul 2026cs.LG cs.CLcited by 0

On the map: LLM agents and tool use → Agent memory systems

Abstract (arXiv)

For local large language model agents, active context is a scarce resource: memory capacity, prefill latency, cache growth, and service objectives all constrain how many input tokens each call can afford. We present BudgetBench, an active-budget protocol and reference harness that treats the per-call input-token budget as the independent variable when comparing memory strategies. Holding the model, task, sampler, and decoding fixed, it sweeps budgets over 2K, 4K, 8K, 16K, and 32K tokens and records quality, budget utilization, latency, and, as a first-class outcome, budget-violation rates. The core contribution is this reusable measurement surface: a swappable MemoryStrategy contract, explicit budget enforcement, deterministic or versioned graders, prompt-audit metadata, and reproducibility artifacts, released at https://github.com/aviskaar/budgetbench. We substantiate the protocol with pilot studies rather than final rankings. Across a local qwen2.5:1.5b pilot (89 items each on SWE-bench Verified and LongBench v2), a hosted 50-item Qwen3 30B-A3B LongBench replication with exact tokenization, and a 500-item LongMemEval oracle study scored by the official GPT-4o evaluator, the harness exposes budget-compliance failures, non-monotonic quality curves, and operating points that single-budget evaluation hides. The budgeted-versus-full-context direction remains unresolved: the local slice is near-null and the hosted replication favors full context in point estimate. We report results transparently, including that the early pilot's tokenizer approximation undercounts some served-model prompts, so its violation rows are tokenizer-approximation diagnostics, not claim-bearing results; all timings are operational diagnostics. The reusable contribution is the protocol, harness, and failure-reporting discipline needed to scale fixed-budget memory-strategy evaluation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Agent memory systems, one of 16 directions in LLM agents and tool use. Builds on 23 of its 28 references that are in the map. First posted on arXiv 1 Jul 2026.

Builds on

23 of 28 references.

Closest ancestors · 10

Same direction, newest first

  1. 2026Less Context, More Accuracy: A Bi-Temporal Memory Engine for LLM Agents Where a Lean Retrieved Context Beats the Full History5 Jun
  2. 2026EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective18 May
  3. 2026LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues12 May
  4. 2026Lightweight LLM Agent Memory with Small Language Models9 Apr
  5. 2026ContextBudget: Budget-Aware Context Management for Long-Horizon Search Agents2 Apr
  6. 2026MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks18 Feb
  7. 2026Learning Query-Aware Budget-Tier Routing for Runtime Agent Memory5 Feb
  8. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions7 Jul
  9. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory28 Apr
  10. 2024Evaluating Very Long-Term Conversational Memory of LLM Agents27 Feb

Other references · 13

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  5. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  6. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  7. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  8. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  9. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 707
  10. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  11. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 348
  12. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 125
1 more
  1. 2026Prompt Compression in the Wild: Measuring Latency, Rate Adherence, and Quality for Faster LLM Inferencecited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Retain or Consolidate? Budget-Dependent Operator Selection for Language Agent Memory
  2. 2026Harnessing LLMs as Agents: What Does It Cost?
  3. 2026Dynamic Budget Allocation for LLM Evaluation under Hard Resource Constraints
  4. 2025TimeBill: Time-Budgeted Inference for Large Language Models
  5. 2025Budget-Aware Tool Use Enables Effective Agent Scaling
  6. 2026What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents
  7. 2025AgentTTS: Large Language Model Agent for Test-time Compute-optimal Scaling Strategy in Complex Tasks
  8. 2026UniACE: A Unified Framework for Evaluating LLM Agentic Capabilities