mlsanityData to 7 Oct 2026

Structured Packing in LLM Training Improves Long Context Utilization

arXiv 2312.17296 ↗First posted 28 Dec 2023cs.CLcited by 17

Not on the map. Why?

Abstract (arXiv)

Recent advancements in long-context large language models have attracted significant attention, yet their practical applications often suffer from suboptimal context utilization. This study investigates structuring training data to enhance semantic interdependence, demonstrating that this approach effectively improves context utilization. To this end, we introduce the Structured Packing for Long Context (SPLiCe) method, which utilizes retrieval to collate mutually relevant documents into long and coherent training examples. We validate SPLiCe empirically across models of varying sizes -- 3B, 7B, and 13B -- achieving improved performance in long-context tasks, such as Qasper and HotpotQA. Remarkably, even brief fine-tuning with SPLiCe is sufficient to realize these benefits. Additionally, SPLiCe effectively mitigates the lost-in-middle phenomenon often observed in large models. Our comprehensive analysis of SPLiCe explores its design choices and reveals intriguing transfer effects; for instance, training on programming code enhances performance on natural language tasks.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 31 of its 58 references that are in the map. First posted on arXiv 28 Dec 2023.

Builds on

31 of 58 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  10. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  11. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  12. 2022Training Compute-Optimal Large Language Modelscited by 2,927
19 more
  1. 2017Billion-scale similarity search with GPUscited by 2,477
  2. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  3. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  4. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  5. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  6. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  7. 2021Ethical and social risks of harm from Language Modelscited by 957
  8. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  9. 2023StarCoder: may the source be with you!cited by 790
  10. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  11. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  12. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  13. 2022Memorizing Transformerscited by 219
  14. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  15. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  16. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  17. 2023In-context Pretraining: Language Modeling Beyond Document Boundariescited by 76
  18. 2023Pre-Training to Learn in Contextcited by 41
  19. 2021The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Designcited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 2; 2026: 4.

17 papers.

Sort:

By area:

  1. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  2. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compressioncited by 84
  3. 2024DAPE: Data-Adaptive Positional Encoding for Length Extrapolationcited by 30
  4. 2024Cofca: A Step-Wise Counterfactual Multi-hop QA benchmarkcited by 26
  5. 2024Analysing The Impact of Sequence Composition on Language Model Pre-Trainingcited by 22
  6. 2024Long Context is Not Long at All: A Prospector of Long-Dependency Data for Large Language Modelscited by 15
  7. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 14
  8. 2025LADM: Long-context Training Data Selection with Attention-based Dependency Measurement for LLMscited by 11
  9. 2024Attribute or Abstain: Large Language Models as Long Document Assistantscited by 10
  10. 2024DAPE V2: Process Attention Score as Feature Map for Length Extrapolationcited by 10
  11. 2025A Survey of LLM × DATAcited by 4
  12. 2024DataSculpt: Crafting Data Landscapes for Long-Context LLMs through Multi-Objective Partitioningcited by 2
5 more
  1. 2026Beyond Rephrasing: Book-Level Organization Improves Synthetic Textbook Data for Mid-Trainingcited by 2
  2. 2024Threshold Filtering Packing for Supervised Fine-Tuning: Training Related Samples within Packscited by 1
  3. 2026OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modelingcited by 1
  4. 2026HSAP: A Hierarchical Sequence-aware Parallelism for Hybrid-Context Generative Modelscited by 0
  5. 2026The Sirens' Song: When Proximal Background Context Overshadows Distant Evidencecited by 0