mlsanityData to 7 Oct 2026

SnapStream: Efficient Long Sequence Decoding on Dataflow Accelerators

arXiv 2511.03092 ↗First posted 5 Nov 2025cs.AI cs.AR cs.DCcited by 2

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

The proliferation of 100B+ parameter Large Language Models (LLMs) with 100k+ context length support have resulted in increasing demands for on-chip memory to support large KV caches. Techniques such as StreamingLLM and SnapKV demonstrate how to control KV cache size while maintaining model accuracy. Yet, these techniques are not commonly used within industrial deployments using frameworks like vLLM or SGLang. The reason is twofold: on one hand, the static graphs and continuous batching methodology employed by these frameworks make it difficult to admit modifications to the standard multi-head attention algorithm, while on the other hand, the accuracy implications of such techniques on modern instruction-following and reasoning models are not well understood, obfuscating the need for implementing these techniques. In this paper, we explore these accuracy implications on Llama-3.1-8B-Instruct and DeepSeek-R1, and develop SnapStream, a KV cache compression method that can be deployed at scale. We demonstrate the efficacy of SnapStream in a 16-way tensor-parallel deployment of DeepSeek-671B on SambaNova SN40L accelerators running at 128k context length and up to 1832 tokens per second in a real production setting. SnapStream enables 4× improved on-chip memory usage and introduces minimal accuracy degradation on LongBench-v2, AIME24 and LiveCodeBench. To the best of our knowledge, this is the first implementation of sparse KV attention techniques deployed in a production inference system with static graphs and continuous batching.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 30 of its 45 references that are in the map. First posted on arXiv 5 Nov 2025.

Builds on

30 of 45 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference11 Mar
  2. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  3. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2020Longformer: The Long-Document Transformercited by 3,362
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  8. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  9. 2020Reformer: The Efficient Transformercited by 1,658
  10. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  11. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  12. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
14 more
  1. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  2. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  3. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  4. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  5. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  6. 2025Kimi K2: Open Agentic Intelligencecited by 523
  7. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  8. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  9. 2024Titans: Learning to Memorize at Test Timecited by 303
  10. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  11. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187
  12. 2025SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inferencecited by 113
  13. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  14. 2024SambaNova SN40L: Scaling the AI Memory Wall with Dataflow and Composition of Expertscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Cross-Family Speculative Prefill: Training-Free Long-Context Compression with Small Draft Modelscited by 2
  2. 2026KV Cache Offloading for Context-Intensive Taskscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025PackKV: Reducing KV Cache Memory Footprint through LLM-Aware Lossy Compression
  2. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
  3. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  4. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
  5. 2025FastKV: Decoupling of Context Reduction and KV Cache Compression for Prefill-Decoding Acceleration
  6. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption
  7. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
  8. 2026VeriCache: Turning Lossy KV Cache into Lossless LLM Inference