mlsanityData to 14 Oct 2026

LampAttention: Look-Ahead Mixed-Precision FlashAttention for Dedicated Accelerators

arXiv 2609.39361 ↗First posted 30 Sep 2026cs.LG cs.NA math.NAcited by 0

Not on the map. Why?

Abstract (arXiv)

While most attention logits can be computed in low precision without degrading numerical stability, current attention kernels fail to exploit this phenomenon. We introduce a novel hardware-algorithm co-design in the form of mixed-precision FlashAttention. Our method accumulates key-query products and evaluates their exponentials in 8-bit formats, then adaptively identifies sensitive sub-blocks and recomputes them in 16-bit formats. We propose the specifications for a dedicated accelerator capable of executing this pipeline efficiently. Simulated experiments with Qwen3 and Gemma 3 show that rerouting a selective minority of sub-blocks to high precision is sufficient to recover the baseline model performance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 28 of its 37 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

28 of 37 references.

References · 28

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2025Qwen3 Technical Reportcited by 8,820
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  9. 2020Longformer: The Long-Document Transformercited by 3,413
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  11. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  12. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
16 more
  1. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  2. 2025Gemma 3 Technical Reportcited by 1,824
  3. 2020Reformer: The Efficient Transformercited by 1,668
  4. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  6. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  7. 2017Mixed Precision Trainingcited by 1,156
  8. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  9. 2015Deep Learning with Limited Numerical Precisioncited by 376
  10. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 232
  11. 2022FP8 Formats for Deep Learningcited by 196
  12. 2026FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scalingcited by 66
  13. 2025Log-Linear Attentioncited by 40
  14. 2025Numerical stability analysis of large language modelscited by 9
  15. 2025Mixed precision accumulation for neural network inference guided by componentwise forward error analysiscited by 6
  16. 2026LAMP: Look-Ahead Mixed-Precision Inference of Large Language Modelscited by 1

Built on by

Not cited yet.