mlsanityData to 14 Oct 2026

Fewer Truncations Improve Language Modeling

arXiv 2404.10830 ↗First posted 16 Apr 2024cs.CL cs.AI cs.LGcited by 35

Not on the map. Why?

Abstract (arXiv)

In large language model training, input documents are typically concatenated together and then split into sequences of equal length to avoid padding tokens. Despite its efficiency, the concatenation approach compromises data integrity -- it inevitably breaks many documents into incomplete pieces, leading to excessive truncations that hinder the model from learning to compose logically coherent and factually consistent content that is grounded on the complete context. To address the issue, we propose Best-fit Packing, a scalable and efficient method that packs documents into training sequences through length-aware combinatorial optimization. Our method completely eliminates unnecessary truncations while retaining the same training efficiency as concatenation. Empirical results from both text and code pre-training show that our method achieves superior performance (e.g., relatively +4.7% on reading comprehension; +16.8% in context following; and +9.2% on program synthesis), and reduces closed-domain hallucination effectively by up to 58.3%.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Builds on 43 of its 60 references that are in the map. First posted on arXiv 16 Apr 2024.

Builds on

43 of 60 references.

References · 43

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2021Program Synthesis with Large Language Modelscited by 3,637
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  12. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
31 more
  1. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  3. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  4. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  7. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  8. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  9. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,292
  10. 2015Teaching Machines to Read and Comprehendcited by 1,049
  11. 2021Ethical and social risks of harm from Language Modelscited by 966
  12. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  13. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  14. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  15. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  16. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  17. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  18. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  19. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
  20. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 458
  21. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 407
  22. 2023Larger language models do in-context learning differentlycited by 394
  23. 2018QuAC : Question Answering in Contextcited by 381
  24. 2022The Stack: 3 TB of permissively licensed source codecited by 359
  25. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 356
  26. 2022Prompting GPT-3 To Be Reliablecited by 260
  27. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 236
  28. 2023Inverse Scaling: When Bigger Isn't Bettercited by 207
  29. 2023In-context Pretraining: Language Modeling Beyond Document Boundariescited by 79
  30. 2023Data-Juicer: A One-Stop Data Processing System for Large Language Modelscited by 52
  31. 2023A Static Evaluation of Code Completion by Large Language Modelscited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 11; 2026: 14.

35 papers.

Sort:

By area:

  1. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  2. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
  3. 2024DeepSeek-V3 Technical Reportcited by 811
  4. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 467
  5. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolutioncited by 263
  6. 2024How to Train Long-Context Language Models (Effectively)cited by 146
  7. 2026Qwen3-Coder-Next Technical Reportcited by 71
  8. 2025Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Modelscited by 54
  9. 2024DAPE: Data-Adaptive Positional Encoding for Length Extrapolationcited by 31
  10. 2024Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculumcited by 26
  11. 2025Metadata Conditioning Accelerates Language Model Pre-trainingcited by 22
  12. 2026SWE-Universe: Scale Real-World Verifiable Environments to Millionscited by 19
23 more
  1. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 15
  2. 2024DAPE V2: Process Attention Score as Feature Map for Length Extrapolationcited by 10
  3. 2025Citrus-V: Advancing Medical Foundation Models with Unified Medical Image Grounding for Clinical Reasoningcited by 8
  4. 2025Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Expertscited by 6
  5. 2025SkyLadder: Better and Faster Pretraining via Context Window Schedulingcited by 6
  6. 2025How Well Can General Vision-Language Models Learn Medicine By Watching Public Educational Videos?cited by 6
  7. 2024From promise to practice: realizing high-performance decentralized trainingcited by 4
  8. 2025A Survey of LLM × DATAcited by 4
  9. 2025K2-V2: A 360-Open, Reasoning-Enhanced LLMcited by 3
  10. 2026Beyond Rephrasing: Book-Level Organization Improves Synthetic Textbook Data for Mid-Trainingcited by 3
  11. 2024Beyond Fixed Length: Bucket Pre-training is All You Needcited by 2
  12. 2026t0: A Time-Series Foundation Model for Forecasting with Contextcited by 2
  13. 2026Nonparametric Teaching of Attention Learnerscited by 1
  14. 2024DEPTH: Discourse Education through Pre-Training Hierarchicallycited by 0
  15. 2025Effect of Document Packing on the Latent Multi-Hop Reasoning Capabilities of Large Language Modelscited by 0
  16. 2026Whole-Piece Training for Symbolic Music Language Models via Full-Horizon Compressed Recurrencecited by 0
  17. 2026Mellum2 Technical Reportcited by 0
  18. 2026TuringLLM: Efficiently Scaling Foundation Models Toward Physical AIcited by 0
  19. 2026MemoryWalker: Stop Training Agents on Contexts They Never Sawcited by 0
  20. 2026Rethinking Length-Based Training: Batch Composition and Loss Normalization in Speech Token Language Modelscited by 0
  21. 2026Linear RNN Scaling Laws: When Longer Sequences Beat More Sequencescited by 0
  22. 2026IronLLM: Forging Compact Edge-Native Language Models for Real-Time Embodied Intelligencecited by 0
  23. 2026Sequence Composition in EHR Foundation Model Pretrainingcited by 0