# MoSKA: Mixture of Shared KV Attention for Efficient Long-Sequence LLM Inference

arXiv: https://arxiv.org/abs/2511.06010 · HTML: https://mlsanity.com/paper/2511.06010/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 14 references that are in the map.
- First posted on arXiv 8 Nov 2025.

## Builds on (most cited first, 13 of 13)

- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [Prompt Cache: Modular Attention Reuse for Low-Latency Inference](https://mlsanity.com/paper/2311.04934/) (2023)
- [FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving](https://mlsanity.com/paper/2501.01005/) (2025)
- [MoBA: Mixture of Block Attention for Long-Context LLMs](https://mlsanity.com/paper/2502.13189/) (2025)
- [ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition](https://mlsanity.com/paper/2402.15220/) (2024)
- [EPIC: Efficient Position-Independent Caching for Serving Large Language Models](https://mlsanity.com/paper/2410.15332/) (2024)
- [Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks](https://mlsanity.com/paper/2412.15605/) (2024)
- [BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batching](https://mlsanity.com/paper/2412.03594/) (2024)
- [MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism](https://mlsanity.com/paper/2504.02263/) (2025)
- [LongHeads: Multi-Head Attention is Secretly a Long Context Processor](https://mlsanity.com/paper/2402.10685/) (2024)
- [Efficient Heterogeneous Large Language Model Decoding with Model-Attention Disaggregation](https://mlsanity.com/paper/2405.01814/) (2024)
- [Forecasting LLM Inference Performance via Hardware-Agnostic Analytical Modeling](https://mlsanity.com/paper/2508.00904/) (2025)

## Built on by (most cited first, 1 of 1)

- [MEPIC: Memory Efficient Position Independent Caching for LLM Serving](https://mlsanity.com/paper/2512.16822/) (2025)

## Abstract (arXiv)

The escalating context length in Large Language Models (LLMs) creates a severe performance bottleneck around the Key-Value (KV) cache, whose memory-bound nature leads to significant GPU under-utilization. This paper introduces Mixture of Shared KV Attention (MoSKA), an architecture that addresses this challenge by exploiting the heterogeneity of context data. It differentiates between per-request unique and massively reused shared sequences. The core of MoSKA is a novel Shared KV Attention mechanism that transforms the attention on shared data from a series of memory-bound GEMV operations into a single, compute-bound GEMM by batching concurrent requests. This is supported by an MoE-inspired sparse attention strategy that prunes the search space and a tailored Disaggregated Infrastructure that specializes hardware for unique and shared data. This comprehensive approach demonstrates a throughput increase of up to 538.7x over baselines in workloads with high context sharing, offering a clear architectural path toward scalable LLM inference.