mlsanityData to 14 Oct 2026

TileMix: Tile-Centric Mixed-Precision Attention for LLM Inference Acceleration

arXiv 2608.17336 ↗First posted 18 Aug 2026cs.AIcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Long-context prefill in large language models (LLMs) incurs substantial computation and memory traffic because dense self-attention computes quadratic query-key scores. Existing methods either use a uniform low-precision path or select token interactions, leaving spatial precision routing over hardware-aligned score tiles outside fused dense attention. We introduce TileMix, a tile-centric precision-routing kernel that makes numerical precision an executable spatial decision over score-tile groups within fused dense attention. TileMix partitions the attention matrix into hardware-aligned score tiles, packs routing decisions into compact bitmasks, and dispatches each tile group through FP16 or INT8 score computation while both paths update a shared online-softmax state. Scalable precision grouping lets each routing bit govern multiple adjacent key tiles, preserving hardware-aligned compute tiles and compact metadata at long contexts. By routing all legal tile groups, TileMix preserves dense token connectivity, requires no training, and supports grouped-query attention, variable-length batches, and INT8 key/value caches. Across LongEval, LV-Eval, and A100 prefill benchmarks on LLaMA, Qwen, and Vicuna, TileMix recovers long-context quality lost under uniform INT8 and improves prefill throughput over FP16, yielding a controllable accuracy-efficiency frontier across model families. The implementation is available at https://github.com/HanzhiZhang-Ulrica/TileMix.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 23 of its 36 references that are in the map. First posted on arXiv 18 Aug 2026.

Builds on

23 of 36 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration3 Oct

Other references · 22

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  3. 2020Longformer: The Long-Document Transformercited by 3,413
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  5. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  6. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  7. 2020Reformer: The Efficient Transformercited by 1,668
  8. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  9. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  10. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  11. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 478
  12. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 471
10 more
  1. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  2. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  3. 2019Adaptively Sparse Transformerscited by 155
  4. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 123
  5. 2024LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256Kcited by 81
  6. 2024ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residualscited by 36
  7. 2023FP8 versus INT8 for efficient deep learning inferencecited by 31
  8. 2024INT-FlashAttention: Enabling Flash Attention for INT8 Quantizationcited by 9
  9. 2025FlashMLA-ETAP: Efficient Transpose Attention Pipeline for Accelerating MLA Inference on NVIDIA H20 GPUscited by 5
  10. 2026ThriftAttention: Selective Mixed Precision for Long-Context FP4 Attentioncited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026PackInfer: Compute- and I/O-Efficient Attention for Batched LLM Inference
  2. 2025LMDeploy Accelerates Mixed-Precision LLM Inference with TurboMind
  3. 2026CSAttention: Centroid-Scoring Attention for Accelerating LLM Inference
  4. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving
  5. 2025MoSKA: Mixture of Shared KV Attention for Efficient Long-Sequence LLM Inference
  6. 2025SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefilling
  7. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
  8. 2025Progressive Sparse Attention: Algorithm and System Co-design for Efficient Attention in LLM Serving