# Enhancing Training Efficiency Using Packing with Flash Attention

arXiv: https://arxiv.org/abs/2407.09105 · HTML: https://mlsanity.com/paper/2407.09105/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 16 references that are in the map.
- First posted on arXiv 12 Jul 2024.

## Builds on (most cited first, 5 of 5)

- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [The Stack: 3 TB of permissively licensed source code](https://mlsanity.com/paper/2211.15533/) (2022)
- [Analysing The Impact of Sequence Composition on Language Model Pre-Training](https://mlsanity.com/paper/2402.13991/) (2024)

## Built on by (most cited first, 12 of 12)

- [MAGI-1: Autoregressive Video Generation at Scale](https://mlsanity.com/paper/2505.13211/) (2025)
- [NeoBERT: A Next-Generation BERT](https://mlsanity.com/paper/2502.19587/) (2025)
- [CursorCore: Assist Programming through Aligning Anything](https://mlsanity.com/paper/2410.07002/) (2024)
- [MojoBench: Language Modeling and Benchmarks for Mojo](https://mlsanity.com/paper/2410.17736/) (2024)
- [Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignment](https://mlsanity.com/paper/2412.07894/) (2024)
- [Small LLMs for Medical NLP: a Systematic Analysis of Few-Shot, Constraint Decoding, Fine-Tuning and Continual Pre-Training in Italian](https://mlsanity.com/paper/2602.17475/) (2026)
- [Hierarchical Balance Packing: Towards Efficient Supervised Fine-tuning for Long-Context LLM](https://mlsanity.com/paper/2503.07680/) (2025)
- [Revisiting Parameter Server in LLM Post-Training](https://mlsanity.com/paper/2601.19362/) (2026)
- [One-Pass to Reason: Token Duplication and Block-Sparse Mask for Efficient Fine-Tuning on Multi-Turn Reasoning](https://mlsanity.com/paper/2504.18246/) (2025)
- [LightRetriever: A LLM-based Text Retrieval Architecture with Extremely Faster Query Inference](https://mlsanity.com/paper/2505.12260/) (2025)
- [FlashCP: Load-Balanced Communication-Efficient Context Parallelism for LLM Training](https://mlsanity.com/paper/2606.08476/) (2026)
- [Shuttling Compiler for Trapped-Ion Quantum Computers Based on Fine-Tuned Large Language Models](https://mlsanity.com/paper/2512.18021/) (2025)

## Abstract (arXiv)

Padding is often used in tuning LLM models by adding special tokens to shorter training examples to match the length of the longest sequence in each batch. While this ensures uniformity for batch processing, it introduces inefficiencies by including irrelevant padding tokens in the computation and wastes GPU resources. Hugging Face SFT trainer has always offered the option to use packing to combine multiple training examples, allowing for maximal utilization of GPU resources. However, up till now, it did not offer proper masking of each packed training example. This capability has been added to Hugging Face Transformers 4.44. We analyse this new feature and show the benefits across different variations of packing.