mlsanityData to 7 Oct 2026

Data Engineering for Scaling Language Models to 128K Context

arXiv 2402.10171 ↗First posted 15 Feb 2024cs.CL cs.AIcited by 183

Not on the map. Why?

Cited by 26 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We study the continual pretraining recipe for scaling language models' context lengths to 128K, with a focus on data engineering. We hypothesize that long context modeling, in particular textitthe ability to utilize information at arbitrary input locations, is a capability that is mostly already acquired through large-scale pretraining, and that this capability can be readily extended to contexts substantially longer than seen during training~(e.g., 4K to 128K) through lightweight continual pretraining on appropriate data mixture. We investigate the textitquantity and textitquality of the data for continual pretraining: (1) for quantity, we show that 500 million to 5 billion tokens are enough to enable the model to retrieve information anywhere within the 128K context; (2) for quality, our results equally emphasize textitdomain balance and textitlength upsampling. Concretely, we find that naively upsampling longer data on certain domains like books, a common practice of existing work, gives suboptimal performance, and that a balanced domain mixture is important. We demonstrate that continual pretraining of the full model on 1B-5B tokens of such data is an effective and affordable strategy for scaling the context length of language models to 128K. Our recipe outperforms strong open-source long-context models and closes the gap to frontier models like GPT-4 128K.

Key facts. Cited by 183 papers in the map (data through 7 Oct 2026). Builds on 30 of its 40 references that are in the map. First posted on arXiv 15 Feb 2024.

Builds on

30 of 40 references.

References · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  9. 2023Qwen Technical Reportcited by 3,273
  10. 2023Mistral 7Bcited by 3,065
  11. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  12. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
18 more
  1. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  2. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  3. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  4. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  5. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  6. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  7. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  8. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  9. 2023Llemma: An Open Language Model For Mathematicscited by 385
  10. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  11. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 239
  12. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  13. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
  14. 2023CodePlan: Repository-level Coding using LLMs and Planningcited by 139
  15. 2021Sequence Parallelism: Long Sequence Training from System Perspectivecited by 125
  16. 2022A Length-Extrapolatable Transformercited by 123
  17. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 100
  18. 2023Peek Across: Improving Multi-Document Modeling via Cross-Document Question-Answeringcited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 83; 2025: 65; 2026: 35.

183 papers.

Sort:

By area:

  1. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  2. 2024Yi: Open Foundation Models by 01.AIcited by 706
  3. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 656
  4. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  5. 2024Long Context Transfer from Language to Visioncited by 502
  6. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
  7. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  8. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  9. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  10. 2024Long-context LLMs Struggle with Long In-context Learningcited by 285
  11. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 275
  12. 2024HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Modelscited by 273
38 more
  1. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  2. 2024Many-Shot In-Context Learningcited by 203
  3. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
  4. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 142
  5. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  6. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  7. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 126
  8. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  9. 2025Context Length Alone Hurts LLM Performance Despite Perfect Retrievalcited by 118
  10. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 116
  11. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecturecited by 100
  12. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Headscited by 99
  13. 2024Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Eracited by 97
  14. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 93
  15. 2024Stable LM 2 1.6B Technical Reportcited by 86
  16. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
  17. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compressioncited by 84
  18. 2024RandAR: Decoder-only Autoregressive Visual Generation in Random Orderscited by 82
  19. 2024Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalizationcited by 78
  20. 2024Why Does the Effective Context Length of LLMs Fall Short?cited by 60
  21. 2024Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Modelscited by 53
  22. 2024ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilitiescited by 52
  23. 2024Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformerscited by 51
  24. 2025M+: Extending MemoryLLM with Scalable Long-Term Memorycited by 51
  25. 2024LongEmbed: Extending Embedding Models for Long Context Retrievalcited by 47
  26. 2024Challenges in Deploying Long-Context Transformers: A Theoretical Peak Performance Analysiscited by 44
  27. 2024GraphReader: Building Graph-based Agent to Enhance Long-Context Abilities of Large Language Modelscited by 42
  28. 2025RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotationscited by 42
  29. 2024SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Modelscited by 41
  30. 2025Scaling Video-Language Models to 10K Frames via Hierarchical Differential Distillationcited by 33
  31. 2024DAPE: Data-Adaptive Positional Encoding for Length Extrapolationcited by 30
  32. 2024MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projectioncited by 29
  33. 2024Large Language Models Can Self-Improve in Long-context Reasoningcited by 29
  34. 2025LongProc: Benchmarking Long-Context Language Models on Long Procedural Generationcited by 28
  35. 2024Efficient Continual Pre-training by Mitigating the Stability Gapcited by 25
  36. 2024LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Modelscited by 23
  37. 2024Long Context Alignment with Short Instructions and Synthesized Positionscited by 22
  38. 2025LongRoPE2: Near-Lossless LLM Context Window Scalingcited by 21

50 most cited of 183 All 183 (JSON)