mlsanityData to 14 Oct 2026

AsyncTLS: Efficient Generative LLM Inference with Asynchronous Two-level Sparse Attention

arXiv 2604.07815 ↗First posted 9 Apr 2026cs.CLcited by 1

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Long-context inference in LLMs faces the dual challenges of quadratic attention complexity and prohibitive KV cache memory. While token-level sparse attention offers superior accuracy, its indexing overhead is costly; block-level methods improve efficiency but sacrifice precision. We propose AsyncTLS, a hierarchical sparse attention system that combines coarse-grained block filtering with fine-grained token selection to balance accuracy and efficiency, coupled with an asynchronous offloading engine that overlaps KV cache transfers with computation via temporal locality exploitation. Evaluated on Qwen3 and GLM-4.7-Flash across GQA, and MLA architectures, AsyncTLS achieves accuracy comparable to full attention while delivering 1.2x - 10.0x operator speedups and 1.3x - 4.7x end-to-end throughput improvements on 48k - 96k contexts.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 27 of its 36 references that are in the map. First posted on arXiv 9 Apr 2026.

Builds on

27 of 36 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMs19 Dec
  2. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
  3. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  4. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  5. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  6. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  7. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr

Other references · 20

  1. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  2. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  4. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  5. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  6. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  7. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  8. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  9. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  10. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 535
  11. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 486
  12. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 467
8 more
  1. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  2. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 365
  3. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
  4. 2021Efficient Attentions for Long Document Summarizationcited by 345
  5. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  6. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 111
  7. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 49
  8. 2024Post-Training Sparse Attention with Double Sparsitycited by 45

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026LongCat Sparse Attention: Taming the Lightning via Streaming-aware Hierarchical Cross-Layer Indexingcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025Long-Context Modeling with Dynamic Hierarchical Sparse Attention for Memory-Constrained LLM Inference
  2. 2025Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference
  3. 2026SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference
  4. 2026Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving
  5. 2026CompKV: Compensation-Aware KV Selection for Long-Context LLM Inference
  6. 2025Progressive Sparse Attention: Algorithm and System Co-design for Efficient Attention in LLM Serving
  7. 2025SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruning
  8. 2026Flux Attention: Context-Aware Hybrid Attention for Efficient LLMs Inference