mlsanityData to 14 Oct 2026

Optimizing Inference in Transformer-Based Models: A Multi-Method Benchmark

arXiv 2509.17894 ↗First posted 22 Sep 2025cs.LGcited by 0

On the map: Diffusion models → Diffusion foundations and sampling

also in 1 more
Abstract (arXiv)

Efficient inference is a critical challenge in deep generative modeling, particularly as diffusion models grow in capacity and complexity. While increased complexity often improves accuracy, it raises compute costs, latency, and memory requirements. This work investigates techniques such as pruning, quantization, knowledge distillation, and simplified attention to reduce computational overhead without impacting performance. The study also explores the Mixture of Experts (MoE) approach to further enhance efficiency. These experiments provide insights into optimizing inference for the state-of-the-art Fast Diffusion Transformer (fast-DiT) model.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 13 of its 18 references that are in the map. First posted on arXiv 22 Sep 2025.

Builds on

13 of 18 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Efficient Diffusion Transformer with Step-wise Dynamic Attention Mediators11 Aug
  2. 2024Scaling Diffusion Transformers to 16 Billion Parameters16 Jul
  3. 2022Scalable Diffusion Models with Transformers19 Dec
  4. 2021Diffusion Models Beat GANs on Image Synthesis11 May

Other references · 9

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  3. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  4. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  6. 2016Pruning Filters for Efficient ConvNetscited by 1,143
  7. 2023Rethinking FID: Towards a Better Evaluation Metric for Image Generationcited by 310
  8. 2021Generating Images with Sparse Representationscited by 266
  9. 2023FLatten Transformer: Vision Transformer using Focused Linear Attentioncited by 171

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Efficiently Scaling Transformer Inference
  2. 2026MoECa: Aligning Feature Reuse with Expert Decomposition in Diffusion Transformers
  3. 2025Dense2MoE: Restructuring Diffusion Transformer to MoE for Efficient Text-to-Image Generation
  4. 2024Remix-DiT: Mixing Diffusion Transformers for Multi-Expert Denoising
  5. 2025FastCache: Fast Caching for Diffusion Transformer Through Learnable Linear Approximation
  6. 2025ALTER: All-in-One Layer Pruning and Temporal Expert Routing for Efficient Diffusion Generation
  7. 2026DiffSparse: Accelerating Diffusion Transformers with Learned Token Sparsity
  8. 2025Scaling Diffusion Transformers Efficiently via μP