mlsanityData to 14 Oct 2026

Architecting Long-Context LLM Acceleration with Packing-Prefetch Scheduler and Ultra-Large Capacity On-Chip Memories

arXiv 2508.08457 ↗First posted 11 Aug 2025cs.AR cs.ETcited by 2

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Long-context Large Language Model (LLM) inference faces increasing compute bottlenecks as attention calculations scale with context length, primarily due to the growing KV-cache transfer overhead that saturates High Bandwidth Memory (HBM). While prefetching techniques mitigate cache misses by fetching KV data in advance, their spatial and temporal benefits present new opportunities to exploit. This work proposes a packing-prefetch scheduling architecture with monolithic 3D (M3D) back-end-of-line (BEOL) compatible embedded memories with ultra-large on-chip capacity to accelerate long-context LLM inference. Our optimizations demonstrate 8.06x decode speedup and 1.83x overall latency reduction on Llama3.1-8B using TPUv6e-like hardware with additional 512MB BEOL memories over the serial execution. Evaluations of multi-request workloads on TPU-like architectures show 1.7x-2.4x throughput improvement and 1.5x-2.4x HBM bandwidth reduction compared to packing-only methods on Llama3.1-8B and Llama3.1-70B models. With the co-design of packing, prefetching, and BEOL memories, our approach alleviates HBM constraints and enables efficient long-context LLM inference.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 8 of its 12 references that are in the map. First posted on arXiv 11 Aug 2025.

Builds on

8 of 12 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  2. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills31 Aug

Other references · 6

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  4. 2021FLAT: An Optimized Dataflow for Mitigating Attention Bottleneckscited by 39
  5. 2025PRESERVE: Prefetching Model Weights and KV-Cache in Distributed LLM Servingcited by 9
  6. 2025CMOS+X: Stacking Persistent Embedded Memories based on Oxide Transistors upon GPGPU Platformscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025LoPT: Lossless Parallel Tokenization Acceleration for Long Context Inference of Large Language Modelcited by 1
  2. 2026LLMET: Enabling Cross-Layer Evaluation of Emerging M3D Memories for Energy-Efficient LLM Servingcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Accelerating LLM Inference via Dynamic KV Cache Placement in Heterogeneous Memory System
  2. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
  3. 2025Accelerating LLM Inference Throughput via Asynchronous KV Cache Prefetching
  4. 2026MOCAP: Wafer-Scale-Chip-Oriented Memory-Orchestrated Chunked Pipelining Framework for Prefill-Only LLM Inference
  5. 2025Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading
  6. 2026CHIPSMORE: Compute-in-Interconnect and -Memory Chiplets for Multi-Mode Multi-Request LLM Inference Acceleration
  7. 2025Reimagining Memory Access for LLM Inference: Compression-Aware Memory Controller Design
  8. 2025LLaMCAT: Optimizing Large Language Model Inference with Cache Arbitration and Throttling