# ByteTransformer: A High-Performance Transformer Boosted for Variable-Length Inputs

arXiv: https://arxiv.org/abs/2210.03052 · HTML: https://mlsanity.com/paper/2210.03052/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 33 references that are in the map.
- First posted on arXiv 6 Oct 2022.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale](https://mlsanity.com/paper/2207.00032/) (2022)
- [Behavior Sequence Transformer for E-commerce Recommendation in Alibaba](https://mlsanity.com/paper/1905.06874/) (2019)
- [TurboTransformers: An Efficient GPU Serving System For Transformer Models](https://mlsanity.com/paper/2010.05680/) (2020)
- [LightSeq2: Accelerated Training for Transformer-based Models on GPUs](https://mlsanity.com/paper/2110.05722/) (2021)
- [Boosting Distributed Training Performance of the Unpadded BERT Model](https://mlsanity.com/paper/2208.08124/) (2022)
- [Accelerating Encrypted Computing on Intel GPUs](https://mlsanity.com/paper/2109.14704/) (2021)
- [FT-BLAS: A High Performance BLAS Implementation With Online Fault Tolerance](https://mlsanity.com/paper/2104.00897/) (2021)

## Built on by (most cited first, 30 of 32)

- [FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving](https://mlsanity.com/paper/2501.01005/) (2025)
- [A Survey on Efficient Inference for Large Language Models](https://mlsanity.com/paper/2404.14294/) (2024)
- [Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations](https://mlsanity.com/paper/2402.17152/) (2024)
- [Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generation](https://mlsanity.com/paper/2307.15337/) (2023)
- [Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems](https://mlsanity.com/paper/2312.15234/) (2023)
- [MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool](https://mlsanity.com/paper/2406.17565/) (2024)
- [Model Compression and Efficient Inference for Large Language Models: A Survey](https://mlsanity.com/paper/2402.09748/) (2024)
- [Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective](https://mlsanity.com/paper/2410.04466/) (2024)
- [How Far Are We From AGI: Are LLMs All We Need?](https://mlsanity.com/paper/2405.10313/) (2024)
- [A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models](https://mlsanity.com/paper/2410.07265/) (2024)
- [SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Serving](https://mlsanity.com/paper/2408.05235/) (2024)
- [WLB-LLM: Workload-Balanced 4D Parallelism for Large Language Model Training](https://mlsanity.com/paper/2503.17924/) (2025)
- [DynaPipe: Optimizing Multi-task Training through Dynamic Pipelines](https://mlsanity.com/paper/2311.10418/) (2023)
- [The Case for Co-Designing Model Architectures with Hardware](https://mlsanity.com/paper/2401.14489/) (2024)
- [SPIN: Accelerating Large Language Model Inference with Heterogeneous Speculative Models](https://mlsanity.com/paper/2503.15921/) (2025)
- [Sol Video Inference Engine: Agent-Native Full-Stack Acceleration Framework for Efficient Video Generation](https://mlsanity.com/paper/2606.23743/) (2026)
- [Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Experts](https://mlsanity.com/paper/2503.05447/) (2025)
- [MACKO: Sparse Matrix-Vector Multiplication for Low Sparsity](https://mlsanity.com/paper/2511.13061/) (2025)
- [SoD2: Statically Optimizing Dynamic Deep Neural Network](https://mlsanity.com/paper/2403.00176/) (2024)
- [Achieving Peak Performance for Large Language Models: A Systematic Review](https://mlsanity.com/paper/2409.04833/) (2024)
- [MCUBERT: Memory-Efficient BERT Inference on Commodity Microcontrollers](https://mlsanity.com/paper/2410.17957/) (2024)
- [Advances in Pre-trained Language Models for Domain-Specific Text Classification: A Systematic Review](https://mlsanity.com/paper/2510.17892/) (2025)
- [FT K-means: A High-Performance K-means on GPU with Fault Tolerance](https://mlsanity.com/paper/2408.01391/) (2024)
- [ACRoBat: Optimizing Auto-batching of Dynamic Deep Learning at Compile Time](https://mlsanity.com/paper/2305.10611/) (2023)
- [PackMamba: Efficient Processing of Variable-Length Sequences in Mamba training](https://mlsanity.com/paper/2408.03865/) (2024)
- [ClusterFusion: Expanding Operator Fusion Scope for LLM Inference via Cluster-Level Collective Primitive](https://mlsanity.com/paper/2508.18850/) (2025)
- [Breaking Scale Anchoring: Frequency Representation Learning for Accurate High-Resolution Inference from Low-Resolution Training](https://mlsanity.com/paper/2512.05132/) (2025)
- [Breaking MLPerf Training: A Case Study on Optimizing BERT](https://mlsanity.com/paper/2402.02447/) (2024)
- [Exploiting Student Parallelism for Efficient GPU Inference of BERT-like Models in Online Services](https://mlsanity.com/paper/2408.12526/) (2024)
- [Accelerating Sparse Transformer Inference on GPU](https://mlsanity.com/paper/2506.06095/) (2025)

## Abstract (arXiv)

Transformers have become keystone models in natural language processing over the past decade. They have achieved great popularity in deep learning applications, but the increasing sizes of the parameter spaces required by transformer models generate a commensurate need to accelerate performance. Natural language processing problems are also routinely faced with variable-length sequences, as word counts commonly vary among sentences. Existing deep learning frameworks pad variable-length sequences to a maximal length, which adds significant memory and computational overhead. In this paper, we present ByteTransformer, a high-performance transformer boosted for variable-length inputs. We propose a padding-free algorithm that liberates the entire transformer from redundant computations on zero padded tokens. In addition to algorithmic-level optimization, we provide architecture-aware optimizations for transformer functional modules, especially the performance-critical algorithm Multi-Head Attention (MHA). Experimental results on an NVIDIA A100 GPU with variable-length sequence inputs validate that our fused MHA outperforms PyTorch by 6.13x. The end-to-end performance of ByteTransformer for a forward BERT transformer surpasses state-of-the-art transformer frameworks, such as PyTorch JIT, TensorFlow XLA, Tencent TurboTransformer, Microsoft DeepSpeed-Inference and NVIDIA FasterTransformer, by 87%, 131%, 138%, 74% and 55%, respectively. We also demonstrate the general applicability of our optimization methods to other BERT-like models, including ALBERT, DistilBERT, and DeBERTa.