mlsanityData to 7 Oct 2026

Enhancing Training Efficiency Using Packing with Flash Attention

arXiv 2407.09105 ↗First posted 12 Jul 2024cs.LG cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

Padding is often used in tuning LLM models by adding special tokens to shorter training examples to match the length of the longest sequence in each batch. While this ensures uniformity for batch processing, it introduces inefficiencies by including irrelevant padding tokens in the computation and wastes GPU resources. Hugging Face SFT trainer has always offered the option to use packing to combine multiple training examples, allowing for maximal utilization of GPU resources. However, up till now, it did not offer proper masking of each packed training example. This capability has been added to Hugging Face Transformers 4.44. We analyse this new feature and show the benefits across different variations of packing.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 5 of its 16 references that are in the map. First posted on arXiv 12 Jul 2024.

Builds on

5 of 16 references.

References · 5

  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  4. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  5. 2024Analysing The Impact of Sequence Composition on Language Model Pre-Trainingcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 6; 2026: 3.

12 papers.

  1. 2025MAGI-1: Autoregressive Video Generation at Scalecited by 262
  2. 2025NeoBERT: A Next-Generation BERTcited by 14
  3. 2024CursorCore: Assist Programming through Aligning Anythingcited by 6
  4. 2024MojoBench: Language Modeling and Benchmarks for Mojocited by 4
  5. 2026Small LLMs for Medical NLP: a Systematic Analysis of Few-Shot, Constraint Decoding, Fine-Tuning and Continual Pre-Training in Italiancited by 3
  6. 2024Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignmentcited by 2
  7. 2025Hierarchical Balance Packing: Towards Efficient Supervised Fine-tuning for Long-Context LLMcited by 2
  8. 2026Revisiting Parameter Server in LLM Post-Trainingcited by 2
  9. 2025One-Pass to Reason: Token Duplication and Block-Sparse Mask for Efficient Fine-Tuning on Multi-Turn Reasoningcited by 1
  10. 2025LightRetriever: A LLM-based Text Retrieval Architecture with Extremely Faster Query Inferencecited by 1
  11. 2025Shuttling Compiler for Trapped-Ion Quantum Computers Based on Fine-Tuned Large Language Modelscited by 0
  12. 2026FlashCP: Load-Balanced Communication-Efficient Context Parallelism for LLM Trainingcited by 0