# Combating the Memory Walls: Optimization Pathways for Long-Context Agentic LLM Inference

arXiv: https://arxiv.org/abs/2509.09505 · HTML: https://mlsanity.com/paper/2509.09505/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Sep 2025.

## Built on by (most cited first, 7 of 7)

- [Slipstream: Trajectory-Grounded Compaction Validation for Long-Horizon Agents](https://mlsanity.com/paper/2605.08580/) (2026)
- [KernelCraft: Benchmarking for Agentic Close-to-Metal Kernel Generation on Emerging Hardware](https://mlsanity.com/paper/2603.08721/) (2026)
- [Mix-Quant: Quantized Prefilling, Precise Decoding for Agentic LLMs](https://mlsanity.com/paper/2605.20315/) (2026)
- [TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks](https://mlsanity.com/paper/2605.17170/) (2026)
- [HTMA-Net: Towards Multiplication-Avoiding Neural Networks via Hadamard Transform and In-Memory Computing](https://mlsanity.com/paper/2509.23103/) (2025)
- [KernelFlume: Elastic Core-Attention Scaling for Agentic Long-Context Decoding](https://mlsanity.com/paper/2606.29207/) (2026)
- [A First-Principles Theory of Slow Thinking and Active Perception](https://mlsanity.com/paper/2607.08196/) (2026)

## Abstract (arXiv)

LLMs now form the backbone of AI agents across a diverse range of applications, including tool use, command-line interfaces, and web or computer interaction. These agentic LLM inference tasks are fundamentally different from chatbot-focused inference. They often involve much longer context lengths to capture complex and prolonged inputs, such as an entire webpage DOM or complicated tool-call trajectories. This, in turn, generates significant off-chip memory traffic during inference and causes workloads to be constrained by two memory walls, namely the bandwidth wall and the capacity wall, preventing compute units from achieving high utilization. In this paper, we introduce PLENA, a hardware-software co-designed system built around three core optimization pathways. PLENA features a novel flattened systolic-array architecture (Pathway 1) and efficient compute and memory units that support an asymmetric quantization scheme (Pathway 2). It also provides native support for FlashAttention (Pathway 3). In addition, PLENA includes a complete software-hardware stack, consisting of a custom ISA, a compiler, a transaction-level simulator, and an automated design-space exploration flow. Experimental results show that PLENA delivers up to 2.23x and 4.70x higher throughput than the A100 GPU and TPU v6e, respectively, under identical multiplier counts and memory configurations during LLaMA agentic inference. PLENA also achieves up to 4.04x higher energy efficiency than the A100 GPU. The full PLENA system, including its simulator, compiler, ISA, and RTL implementation, will be open-sourced to the research community.