mlsanityData to 14 Oct 2026

FlashFormer: Whole-Model Kernels for Efficient Low-Batch Inference

arXiv 2505.22758 ↗First posted 28 May 2025cs.LG cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

The size and compute characteristics of modern large language models have led to an increased interest in developing specialized kernels tailored for particular training and inference workloads. Existing kernels primarily optimize for compute utilization, targeting the large-batch training and inference settings. However, low-batch inference, where memory bandwidth and kernel launch overheads are significant factors, remains important for many applications of interest such as in edge deployment and latency-sensitive applications. This paper describes FlashFormer, which fuses the entire transformer forward pass into a single kernel for accelerating low-batch inference of large language models. Across various model sizes and quantizations settings, FlashFormer achieves nontrivial speedups compared to existing inference kernels.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 17 of its 28 references that are in the map. First posted on arXiv 28 May 2025.

Builds on

17 of 28 references.

References · 17

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  6. 2020GLU Variants Improve Transformercited by 1,804
  7. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  9. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  10. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  11. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  12. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
5 more
  1. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  2. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 232
  3. 2018Online normalizer calculation for softmaxcited by 162
  4. 2024Mirage: A Multi-Level Superoptimizer for Tensor Programscited by 41
  5. 2023Pre-RMSNorm and Pre-CRMSNorm Transformers: Equivalent and Efficient Pre-LN Transformerscited by 30

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2026FlashSampling: Fast and Memory-Efficient Exact Samplingcited by 3
  2. 2025Learning to Focus: Focal Attention for Selective and Scalable Transformerscited by 2
  3. 2026Weave: Fine-Grained Dynamic SM Scheduling in an MoE Megakernel for Compute-Communication Overlapcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memory
  2. 2024Flash Communication: Reducing Tensor Parallelization Bottleneck for Fast Large Language Model Inference
  3. 2024Highly Optimized Kernels and Fine-Grained Codebooks for LLM Inference on Arm CPUs
  4. 2023FlashDecoding++: Faster Large Language Model Inference on GPUs
  5. 2026Understand and Accelerate Memory Processing Pipeline for Large Language Model Inference
  6. 2024GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments
  7. 2023Dissecting the Runtime Performance of the Training, Fine-tuning, and Inference of Large Language Models
  8. 2024Model Compression and Efficient Inference for Large Language Models: A Survey