mlsanityData to 7 Oct 2026

MeshKV: A Network-on-Chip KV Cache Fabric for Scalable Transformer Decoding Accelerators

arXiv 2609.19207 ↗First posted 16 Sep 2026cs.AR cs.AIcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Autoregressive transformer decoding is constrained by irregular key-value (KV) cache movement on tiled accelerators. Prior compression and DRAM-placement systems still concentrate traffic on centralized memory paths that bottleneck long-context serving. We present MeshKV, a KV cache fabric that moves blocks as packetized flows over a lightweight NoC. It co-designs (i) TaKV affine striping to spread homes and cut hotspot load, (ii) Mare multicast with verified duplicate suppression, and (iii) Pad, which overlaps prefetch, tile multiply, and streaming softmax behind credit-aligned FIFOs. Together they convert bisection back-pressure into useful KV transfer. On our 8x8 FPGA implementation with LLaMA-2-7B and Mistral-7B at 8K-32K, MeshKV reduces interconnect traffic by up to 58%, improves KV bandwidth utilization by 2.1x, and delivers up to 1.9x multi-stream throughput.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 17 of its 24 references that are in the map. First posted on arXiv 16 Sep 2026.

Builds on

17 of 24 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025CXL-SpecKV: A Disaggregated FPGA Speculative KV-Cache for Datacenter LLM Serving11 Dec
  2. 2025TinyServe: Query-Aware Cache Selection for Efficient LLM Serving28 Aug
  3. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 13

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2023Mistral 7Bcited by 3,065
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  7. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  8. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  9. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  10. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 228
  11. 2024FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAscited by 60
  12. 2022DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generationcited by 46
1 more
  1. 2024LLMEasyQuant: Scalable Quantization for Parallel and Distributed LLM Inferencecited by 15

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decoding
  2. 2026Efficient Remote KV Cache Reuse with GPU-native Video Codec
  3. 2026TurboAngle: Near-Lossless KV Cache Compression via Uniform Angle Quantization
  4. 2025Helix Parallelism: Rethinking Sharding Strategies for Interactive Multi-Million-Token LLM Decoding
  5. 2026Stochastic KV Routing: Enabling Adaptive Depth-Wise Cache Sharing
  6. 2025Reconstructing KV Caches with Cross-layer Fusion For Enhanced Transformers
  7. 2025TraCT: Disaggregated LLM Serving with CXL Shared Memory KV Cache at Rack-Scale
  8. 20263DLS: A 3D Logic-Stacked Architecture for Disaggregated LLM Serving