mlsanityData to 13 Oct 2026

ByteTransformer: A High-Performance Transformer Boosted for Variable-Length Inputs

arXiv 2210.03052 ↗First posted 6 Oct 2022cs.LGcited by 32

Not on the map. Why?

Abstract (arXiv)

Transformers have become keystone models in natural language processing over the past decade. They have achieved great popularity in deep learning applications, but the increasing sizes of the parameter spaces required by transformer models generate a commensurate need to accelerate performance. Natural language processing problems are also routinely faced with variable-length sequences, as word counts commonly vary among sentences. Existing deep learning frameworks pad variable-length sequences to a maximal length, which adds significant memory and computational overhead. In this paper, we present ByteTransformer, a high-performance transformer boosted for variable-length inputs. We propose a padding-free algorithm that liberates the entire transformer from redundant computations on zero padded tokens. In addition to algorithmic-level optimization, we provide architecture-aware optimizations for transformer functional modules, especially the performance-critical algorithm Multi-Head Attention (MHA). Experimental results on an NVIDIA A100 GPU with variable-length sequence inputs validate that our fused MHA outperforms PyTorch by 6.13x. The end-to-end performance of ByteTransformer for a forward BERT transformer surpasses state-of-the-art transformer frameworks, such as PyTorch JIT, TensorFlow XLA, Tencent TurboTransformer, Microsoft DeepSpeed-Inference and NVIDIA FasterTransformer, by 87%, 131%, 138%, 74% and 55%, respectively. We also demonstrate the general applicability of our optimization methods to other BERT-like models, including ALBERT, DistilBERT, and DeBERTa.

Key facts. Cited by 32 papers in the map (data through 13 Oct 2026). Builds on 20 of its 33 references that are in the map. First posted on arXiv 6 Oct 2022.

Builds on

20 of 33 references.

References · 20

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2020Language Models are Few-Shot Learnerscited by 32,320
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,223
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,281
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,637
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,472
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,810
  9. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,779
  10. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,171
  11. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,851
  12. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
8 more
  1. 2019BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformercited by 958
  2. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 406
  3. 2019Behavior Sequence Transformer for E-commerce Recommendation in Alibabacited by 129
  4. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71
  5. 2021LightSeq2: Accelerated Training for Transformer-based Models on GPUscited by 19
  6. 2022Boosting Distributed Training Performance of the Unpadded BERT Modelcited by 15
  7. 2021Accelerating Encrypted Computing on Intel GPUscited by 4
  8. 2021FT-BLAS: A High Performance BLAS Implementation With Online Fault Tolerancecited by 3

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 4; 2024: 16; 2025: 9; 2026: 3.

32 papers.

Sort:

By area:

  1. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 253
  2. 2024A Survey on Efficient Inference for Large Language Modelscited by 202
  3. 2024Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendationscited by 185
  4. 2023Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generationcited by 103
  5. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  6. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 67
  7. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  8. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 37
  9. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  10. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 23
  11. 2024SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Servingcited by 18
  12. 2025WLB-LLM: Workload-Balanced 4D Parallelism for Large Language Model Trainingcited by 17
20 more
  1. 2023DynaPipe: Optimizing Multi-task Training through Dynamic Pipelinescited by 12
  2. 2024The Case for Co-Designing Model Architectures with Hardwarecited by 11
  3. 2025SPIN: Accelerating Large Language Model Inference with Heterogeneous Speculative Modelscited by 9
  4. 2026Sol Video Inference Engine: Agent-Native Full-Stack Acceleration Framework for Efficient Video Generationcited by 7
  5. 2025Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Expertscited by 6
  6. 2025MACKO: Sparse Matrix-Vector Multiplication for Low Sparsitycited by 6
  7. 2024SoD2: Statically Optimizing Dynamic Deep Neural Networkcited by 4
  8. 2024Achieving Peak Performance for Large Language Models: A Systematic Reviewcited by 4
  9. 2024MCUBERT: Memory-Efficient BERT Inference on Commodity Microcontrollerscited by 3
  10. 2025Advances in Pre-trained Language Models for Domain-Specific Text Classification: A Systematic Reviewcited by 3
  11. 2024FT K-means: A High-Performance K-means on GPU with Fault Tolerancecited by 2
  12. 2023ACRoBat: Optimizing Auto-batching of Dynamic Deep Learning at Compile Timecited by 1
  13. 2024PackMamba: Efficient Processing of Variable-Length Sequences in Mamba trainingcited by 1
  14. 2025ClusterFusion: Expanding Operator Fusion Scope for LLM Inference via Cluster-Level Collective Primitivecited by 1
  15. 2025Breaking Scale Anchoring: Frequency Representation Learning for Accurate High-Resolution Inference from Low-Resolution Trainingcited by 1
  16. 2024Breaking MLPerf Training: A Case Study on Optimizing BERTcited by 0
  17. 2024Exploiting Student Parallelism for Efficient GPU Inference of BERT-like Models in Online Servicescited by 0
  18. 2025Accelerating Sparse Transformer Inference on GPUcited by 0
  19. 2026HEPTv2: End-to-End Efficient Point Transformer for Charged Particle Reconstructioncited by 0
  20. 2026Sol-H3: Recursive Self-Improvement for MiniMax-H3 Inference Acceleration on Sol-Engine across Cloud and Edgecited by 0