mlsanityData to 14 Oct 2026

SinkRouter: Sink-Aware Routing for Efficient Long-Context Decoding in Large Language and Multimodal Models

arXiv 2604.16883 ↗First posted 18 Apr 2026cs.LG cs.AIcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

In long-context decoding for LLMs and LMMs, attention becomes increasingly memory-bound because each decoding step must load a large amount of KV-cache data from GPU memory. Existing acceleration strategies often trade efficiency for accuracy by relying on heuristic pruning that may discard useful information. At a deeper level, they also tend to indiscriminately preserve all high-scoring tokens, treat early tokens as indispensable anchors, or rely on heuristic head routing, reflecting an insufficient mechanistic understanding of the attention sink phenomenon. In this paper, we show that the attention sink phenomenon corresponds to a stable, reachable, and error-controllable fixed point constructed during training. Based on this insight, we propose SinkRouter, a training-free selective routing framework that detects the sink signal and skips computations that would otherwise produce near-zero output. To translate this mechanism into real-world acceleration, we develop a hardware-aware Triton kernel with block-level branching and Split-K parallelism. We conduct extensive evaluations on a diverse suite of long-context benchmarks, including LongBench, InfiniteBench, CVBench, MileBench, and MMVP, using both text-only and multimodal backbones such as Llama-3.1-8B, Llama-3.1-70B, Yi-9B-200K, LLaVA-1.5-7B, and LLaVA-1.5-13B. Across these settings, SinkRouter consistently improves decoding efficiency while maintaining competitive accuracy, and reaches 2.03x speedup with a 512K context.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 27 of its 34 references that are in the map. First posted on arXiv 18 Apr 2026.

Builds on

27 of 34 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Hardware-Efficient Attention for Fast Decoding27 May
  2. 2025FastKV: Decoupling of Context Reduction and KV Cache Compression for Prefill-Decoding Acceleration3 Feb
  3. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
  4. 2024When Attention Sink Emerges in Language Models: An Empirical View14 Oct
  5. 2024InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference8 Sep
  6. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  7. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference14 Feb
  8. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  9. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 18

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  5. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  6. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  7. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 642
  8. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  9. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  10. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  11. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 313
  12. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 283
6 more
  1. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 252
  2. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  3. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 151
  4. 2024MileBench: Benchmarking MLLMs in Long Contextcited by 88
  5. 2025CVBench: Benchmarking Cross-Video Synergies for Complex Multimodal Reasoningcited by 16
  6. 2026On the Nature of Attention Sink that Shapes Decoding Strategy in Omni-LLMscited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026ART: Attention Run-time Termination for Efficient Large Language Model Decoding
  2. 2026Flux Attention: Context-Aware Hybrid Attention for Efficient LLMs Inference
  3. 2026Elastic Threshold Attention: Learned Contextual Sparsity for Long-Context Decoding
  4. 2026Salca: A Sparsity-Aware Hardware Accelerator for Efficient Long-Context Attention Decoding
  5. 2026Faster Than Flash: Exploiting Attention Sparsity for Efficient Long-Context Decoding
  6. 2025Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding
  7. 2026LycheeDecode: Accelerating Long-Context LLM Inference via Hybrid-Head Sparse Decoding
  8. 2026HiSparse: Scaling Sparse-Attention Decoding with Hierarchical KV Cache Management