mlsanityData to 7 Oct 2026

How to Train Long-Context Language Models (Effectively)

arXiv 2410.02660 ↗First posted 3 Oct 2024cs.CL cs.LGcited by 141

Not on the map. Why?

Abstract (arXiv)

We study continued training and supervised fine-tuning (SFT) of a language model (LM) to make effective use of long-context information. We first establish a reliable evaluation protocol to guide model development -- instead of perplexity or simple needle-in-a-haystack (NIAH) tests, we use a broad set of long-context downstream tasks, and we evaluate models after SFT as this better reveals long-context abilities. Supported by our robust evaluations, we run thorough experiments to decide the data mix for continued pre-training, the instruction tuning dataset, and many other design choices such as position extrapolation. We find that (1) code repositories and books are excellent sources of long data, but it is crucial to combine them with high-quality short-context data; (2) training with a sequence length beyond the evaluation length boosts long-context performance; (3) for SFT, using only short instruction datasets yields strong performance on long-context tasks. Our final model, ProLong-8B, which is initialized from Llama-3 and trained on 40B tokens, demonstrates state-of-the-art long-context performance among similarly sized models at a length of 128K. ProLong outperforms Llama-3.1-8B-Instruct on the majority of long-context tasks despite using only 5% as many tokens during long-context training. Additionally, ProLong can effectively process up to 512K tokens, one of the longest context windows of publicly available LMs.

Key facts. Cited by 141 papers in the map (data through 7 Oct 2026). Builds on 92 of its 109 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

92 of 109 references.

References · 92

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  12. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
38 more
  1. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  4. 2024Qwen2 Technical Reportcited by 2,076
  5. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  6. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  7. 2020Rethinking Attention with Performerscited by 1,584
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  9. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  10. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  11. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  12. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  13. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  14. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  15. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  16. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  17. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  18. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  19. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  20. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  21. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  22. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  23. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  24. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  25. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  26. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 419
  27. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  28. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 381
  29. 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 378
  30. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  31. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  32. 2023Adapting Language Models to Compress Contextscited by 350
  33. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  34. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
  35. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  36. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  37. 2024Long-context LLMs Struggle with Long In-context Learningcited by 285
  38. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243

50 most cited of 92 All 92 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 72; 2026: 55.

141 papers.

Sort:

By area:

  1. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  2. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  3. 2025Scaling Latent Reasoning via Looped Language Modelscited by 145
  4. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 112
  5. 2024Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencentcited by 89
  6. 2026Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Modelscited by 78
  7. 2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 46
  8. 2025QwenLong-L1: Towards Long-Context Large Reasoning Models with Reinforcement Learningcited by 46
  9. 2025mmBERT: A Modern Multilingual Encoder with Annealed Language Learningcited by 39
  10. 2025Seq vs Seq: An Open Suite of Paired Encoders and Decoderscited by 34
  11. 2024Large Language Models Can Self-Improve in Long-context Reasoningcited by 29
  12. 2025Steering off Course: Reliability Challenges in Steering Language Modelscited by 29
38 more
  1. 2025MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughlycited by 29
  2. 2025LongProc: Benchmarking Long-Context Language Models on Long Procedural Generationcited by 28
  3. 2025EuroBERT: Scaling Multilingual Encoders for European Languagescited by 28
  4. 2025Demystifying Domain-adaptive Post-training for Financial LLMscited by 24
  5. 2025LongRoPE2: Near-Lossless LLM Context Window Scalingcited by 21
  6. 2025Chain-of-Thought Matters: Improving Long-Context Language Models with Reasoning Path Supervisioncited by 20
  7. 2025NExtLong: Toward Effective Long-Context Training without Long Documentscited by 19
  8. 2025LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verificationcited by 19
  9. 2025One ruler to measure them all: Benchmarking multilingual long-context language modelscited by 18
  10. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inferencecited by 18
  11. 2025Mid-Training of Large Language Models: A Surveycited by 18
  12. 2025Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMscited by 18
  13. 2025LongFaith: Enhancing Long-Context Reasoning in LLMs with Faithful Synthetic Datacited by 16
  14. 2025SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modelingcited by 16
  15. 2024When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Trainingcited by 15
  16. 2024A Silver Bullet or a Compromise for Full Attention? A Comprehensive Study of Gist Token-based Context Compressioncited by 15
  17. 2025LongReD: Mitigating Short-Text Degradation of Long-Context Large Language Models via Restoration Distillationcited by 14
  18. 2025Token Weighting for Long-Range Language Modelingcited by 14
  19. 2025Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?cited by 13
  20. 2025From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Modelscited by 13
  21. 2025UltraLLaDA: Scaling the Context Length to 128K for Diffusion Large Language Modelscited by 13
  22. 2026LongRLVR: Long-Context Reinforcement Learning Requires Verifiable Context Rewardscited by 13
  23. 2025BLASST: Dynamic BLocked Attention Sparsity via Softmax Thresholdingcited by 12
  24. 2024Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similaritycited by 11
  25. 2024MDCure: A Scalable Pipeline for Multi-Document Instruction-Followingcited by 10
  26. 2025ByteScale: Efficient Scaling of LLM Training with a 2048K Context Length on More Than 12,000 GPUscited by 10
  27. 2025SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimizationcited by 10
  28. 2025Cache Me If You Can: How Many KVs Do You Need for Effective Long-Context LMs?cited by 10
  29. 2025Decoder-Hybrid-Decoder Architecture for Efficient Reasoning with Long Generationcited by 10
  30. 2026Arcee Trinity Large Technical Reportcited by 10
  31. 2025CLIPPER: Compression enables long-context synthetic data generationcited by 9
  32. 2025WildLong: Synthesizing Realistic Long-Context Instruction Data at Scalecited by 9
  33. 2025LongAttn: Selecting Long-context Training Data via Token-level Attentioncited by 9
  34. 2025SOFT: Selective Data Obfuscation for Protecting LLM Fine-tuning against Membership Inference Attackscited by 9
  35. 2025Gated KalmaNet: A Fading Memory Layer Through Test-Time Ridge Regressioncited by 9
  36. 2025AbsenceBench: Language Models Can't Tell What's Missingcited by 8
  37. 2025Infinite Retrieval: Attention Enhanced LLMs in Long-Context Processingcited by 7
  38. 2025Generalizing From Short to Long: Effective Data Synthesis for Long-Context Instruction Tuningcited by 7

50 most cited of 141 All 141 (JSON)