# FastFormers: Highly Efficient Transformer Models for Natural Language Understanding

arXiv: https://arxiv.org/abs/2010.13382 · HTML: https://mlsanity.com/paper/2010.13382/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Oct 2020.

## Built on by (most cited first, 28 of 28)

- [FNet: Mixing Tokens with Fourier Transforms](https://mlsanity.com/paper/2105.03824/) (2021)
- [The Unreasonable Ineffectiveness of the Deeper Layers](https://mlsanity.com/paper/2403.17887/) (2024)
- [Block Pruning For Faster Transformers](https://mlsanity.com/paper/2109.04838/) (2021)
- [Sparse is Enough in Scaling Transformers](https://mlsanity.com/paper/2111.12763/) (2021)
- [Prune Once for All: Sparse Pre-Trained Language Models](https://mlsanity.com/paper/2111.05754/) (2021)
- [A Survey on Model Compression and Acceleration for Pretrained Language Models](https://mlsanity.com/paper/2202.07105/) (2022)
- [Visual Attention Methods in Deep Learning: An In-Depth Survey](https://mlsanity.com/paper/2204.07756/) (2022)
- [HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformers](https://mlsanity.com/paper/2302.09632/) (2023)
- [AccelTran: A Sparsity-Aware Accelerator for Dynamic Inference with Transformers](https://mlsanity.com/paper/2302.14705/) (2023)
- [FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMs](https://mlsanity.com/paper/2308.09723/) (2023)
- [KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillation](https://mlsanity.com/paper/2109.06243/) (2021)
- [SLMRec: Distilling Large Language Models into Small for Sequential Recommendation](https://mlsanity.com/paper/2405.17890/) (2024)
- [On the link between conscious function and general intelligence in humans and machines](https://mlsanity.com/paper/2204.05133/) (2022)
- [FAR: Fourier Aerial Video Recognition](https://mlsanity.com/paper/2203.10694/) (2022)
- [A Theoretical View on Sparsely Activated Networks](https://mlsanity.com/paper/2208.04461/) (2022)
- [MoMA: Momentum Contrastive Learning with Multi-head Attention-based Knowledge Distillation for Histopathology Image Analysis](https://mlsanity.com/paper/2308.16561/) (2023)
- [Large Language Models Are Overparameterized Text Encoders](https://mlsanity.com/paper/2410.14578/) (2024)
- [SwiftPruner: Reinforced Evolutionary Pruning for Efficient Ad Relevance](https://mlsanity.com/paper/2209.00625/) (2022)
- [Investigating Neural Architectures by Synthetic Dataset Design](https://mlsanity.com/paper/2204.11045/) (2022)
- [SAMP: A Model Inference Toolkit of Post-Training Quantization for Text Processing via Self-Adaptive Mixed-Precision](https://mlsanity.com/paper/2209.09130/) (2022)
- [AttMEMO : Accelerating Transformers with Memoization on Big Memory Systems](https://mlsanity.com/paper/2301.09262/) (2023)
- [Transformer with Memory Replay](https://mlsanity.com/paper/2205.09869/) (2022)
- [Input Conditioned Layer Dropping in Speech Foundation Models](https://mlsanity.com/paper/2507.07954/) (2025)
- [Building an Efficiency Pipeline: Commutativity and Cumulativeness of Efficiency Operators for Transformers](https://mlsanity.com/paper/2208.00483/) (2022)
- [Confidence Preservation Property in Knowledge Distillation Abstractions](https://mlsanity.com/paper/2401.11365/) (2024)
- [On Importance of Pruning and Distillation for Efficient Low Resource NLP](https://mlsanity.com/paper/2409.14162/) (2024)
- [On Multilingual Encoder Language Model Compression for Low-Resource Languages](https://mlsanity.com/paper/2505.16956/) (2025)
- [Restoring Rhythm: Punctuation Restoration Using Transformer Models for Bangla, A Low-Resource Language](https://mlsanity.com/paper/2507.18448/) (2025)

## Abstract (arXiv)

Transformer-based models are the state-of-the-art for Natural Language Understanding (NLU) applications. Models are getting bigger and better on various tasks. However, Transformer models remain computationally challenging since they are not efficient at inference-time compared to traditional approaches. In this paper, we present FastFormers, a set of recipes to achieve efficient inference-time performance for Transformer-based models on various NLU tasks. We show how carefully utilizing knowledge distillation, structured pruning and numerical optimization can lead to drastic improvements on inference efficiency. We provide effective recipes that can guide practitioners to choose the best settings for various NLU tasks and pretrained models. Applying the proposed recipes to the SuperGLUE benchmark, we achieve from 9.8x up to 233.9x speed-up compared to out-of-the-box models on CPU. On GPU, we also achieve up to 12.4x speed-up with the presented methods. We show that FastFormers can drastically reduce cost of serving 100 million requests from 4,223 USD to just 18 USD on an Azure F16s_v2 instance. This translates to a sustainable runtime by reducing energy consumption 6.9x - 125.8x according to the metrics used in the SustaiNLP 2020 shared task.