mlsanityData to 7 Oct 2026

Akashic: A Low-Overhead LLM Inference Service with MemAttention

arXiv 2607.05708 ↗First posted 7 Jul 2026cs.AIcited by 0

On the map: LLM agents and tool use → Agent memory systems

Abstract (arXiv)

Recent LLM-based agent systems continuously accumulate context across multi-turn interactions, tool invocations, and cross-session workflows. Replaying the full history for every request quickly becomes impractical: long contexts increase prefill cost, may exceed context limits, and often bury task-relevant evidence in irrelevant content, degrading both serving efficiency and output quality. We propose Akashic, a low-overhead memory system built around MemAttention, which organizes context into bounded chunks and models semantic relationships across chunks, preserving cross-chunk evidence without repeatedly rewriting the full history. Akashic further applies hardware-software co-designed memory placement to co-locate likely co-retrieved chunks, reducing retrieval fragmentation and I/O overhead. Across four representative workloads and three model sizes, Akashic improves task accuracy by up to 10.2 points, throughput by up to 1.21x, and sustainable request rate by up to 1.88x over strong prior memory baselines.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Agent memory systems, one of 16 directions in LLM agents and tool use. Builds on 22 of its 43 references that are in the map. First posted on arXiv 7 Jul 2026.

Builds on

22 of 43 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory28 Apr
  2. 2025In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agents11 Mar
  3. 2025A-MEM: Agentic Memory for LLM Agents17 Feb
  4. 2024Evaluating Very Long-Term Conversational Memory of LLM Agents27 Feb

Other references · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  8. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  9. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  10. 2023Large Language Modelscited by 774
  11. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 616
  12. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
6 more
  1. 2024Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encodingcited by 72
  2. 2021FreshDiskANN: A Fast and Accurate Graph-Based ANN Index for Streaming Similarity Searchcited by 36
  3. 2021SPANN: Highly-efficient Billion-scale Approximate Nearest Neighbor Searchcited by 35
  4. 2024Starling: An I/O-Efficient Disk-Resident Graph Index Framework for High-Dimensional Vector Similarity Search on Data Segmentcited by 16
  5. 2026RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttentioncited by 6
  6. 2025LCIRC: A Recurrent Compression Approach for Efficient Long-form Context and Query Dependent Modeling in LLMscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026MemGym: a Long-Horizon Memory Environment for LLM Agents
  2. 2026ContextWeaver: Selective and Dependency-Structured Memory Construction for LLM Agents
  3. 2026Less Context, More Accuracy: A Bi-Temporal Memory Engine for LLM Agents Where a Lean Retrieved Context Beats the Full History
  4. 2026MemForest: An Efficient Agent Memory System with Hierarchical Temporal Indexing
  5. 2026SimpleMem: Efficient Lifelong Memory for LLM Agents
  6. 2026Sutradhara: An Intelligent Orchestrator-Engine Co-design for Tool-based Agentic Inference
  7. 2026Persistent Memory in Multi-Agent LLM Inference: What It Costs, What It Buys, and When You Can Tell
  8. 2026Stateful Inference for Low-Latency Multi-Agent Tool Calling