mlsanityData to 14 Oct 2026

Diagonal-Tiled Mixed-Precision Attention for Efficient Low-Bit MXFP Inference

arXiv 2604.03950 ↗First posted 5 Apr 2026cs.LG cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Transformer-based large language models (LLMs) have demonstrated remarkable performance across a wide range of real-world tasks, but their inference cost remains prohibitively high due to the quadratic complexity of attention and the memory bandwidth limitations of high-precision operations. In this work, we present a low-bit mixed-precision attention kernel using the microscaling floating-point (MXFP) data format, utilizing the computing capability on next-generation GPU architectures. Our Diagonal-Tiled Mixed-Precision Attention (DMA) incorporates two kinds of low-bit computation at the tiling-level, and is a delicate fused kernel implemented using Triton, exploiting hardware-level parallelism and memory efficiency to enable fast and efficient inference without compromising model performance. Extensive empirical evaluations on NVIDIA B200 GPUs show that our kernel maintains generation quality with negligible degradation, and meanwhile achieves significant speedup by kernel fusion. We release our code at https://github.com/yifu-ding/MP-Sparse-Attn.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 16 of its 20 references that are in the map. First posted on arXiv 5 Apr 2026.

Builds on

16 of 20 references.

References · 16

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  6. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  7. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  8. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  9. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 160
  10. 2023Microscaling Data Formats for Deep Learningcited by 152
  11. 2022On The Computational Complexity of Self-Attentioncited by 139
  12. 2025SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inferencecited by 120
4 more
  1. 2024SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantizationcited by 114
  2. 2025SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Trainingcited by 66
  3. 2025Efficient Attention Mechanisms for Large Language Models: A Surveycited by 22
  4. 2024TurboAttention: Efficient Attention Approximation For High Throughputs LLMscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026DraftAttention2: Fast Video Diffusion with Low-Resolution-Guided Mixed-Precision Attentioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Ragged Paged Attention: A High-Performance and Flexible LLM Inference Kernel for TPU
  2. 2026Fast NF4 Dequantization Kernels for Large Language Model Inference
  3. 2024Efficient Arbitrary Precision Acceleration for Large Language Models on GPU Tensor Cores
  4. 2026Understand and Accelerate Memory Processing Pipeline for Large Language Model Inference
  5. 2025The Anatomy of a Triton Attention Kernel
  6. 2026Learning how to Forget: Fine-tuning for Long-Context Sparse Attention
  7. 2023Fast and Efficient 2-bit LLM Inference on GPU: 2/4/16-bit in a Weight Matrix with Asynchronous Dequantization
  8. 2025APT-LLM: Exploiting Arbitrary-Precision Tensor Core Computing for LLM Acceleration