mlsanityData to 7 Oct 2026

From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models

arXiv 2504.06214 ↗First posted 8 Apr 2025cs.CL cs.AI cs.LGcited by 13

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Long-context capabilities are essential for a wide range of applications, including document and video understanding, in-context learning, and inference-time scaling, all of which require models to process and reason over long sequences of text and multimodal data. In this work, we introduce a efficient training recipe for building ultra-long context LLMs from aligned instruct model, pushing the boundaries of context lengths from 128K to 1M, 2M, and 4M tokens. Our approach leverages efficient continued pretraining strategies to extend the context window and employs effective instruction tuning to maintain the instruction-following and reasoning abilities. Our UltraLong-8B, built on Llama3.1-Instruct with our recipe, achieves state-of-the-art performance across a diverse set of long-context benchmarks. Importantly, models trained with our approach maintain competitive performance on standard benchmarks, demonstrating balanced improvements for both long and short context tasks. We further provide an in-depth analysis of key design choices, highlighting the impacts of scaling strategies and data composition. Our findings establish a robust framework for efficiently scaling context lengths while preserving general model capabilities. We release all model weights at: https://ultralong.github.io/.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 31 of its 43 references that are in the map. First posted on arXiv 8 Apr 2025.

Builds on

31 of 43 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Advancing LLM Reasoning Generalists with Preference Trees2 Apr

Other references · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  10. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  11. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  12. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
18 more
  1. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  2. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  3. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  4. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  5. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 251
  6. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  7. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  8. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  9. 2025Cosmos-Reason1: From Physical Common Sense To Embodied Reasoningcited by 159
  10. 2025Qwen2.5-1M Technical Reportcited by 158
  11. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  12. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 124
  13. 2024LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256Kcited by 80
  14. 2024AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modelingcited by 76
  15. 2024ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilitiescited by 52
  16. 2024A Controlled Study on Long Context Extension and Generalization in LLMscited by 26
  17. 2024TransformerFAM: Feedback attention is working memorycited by 21
  18. 2024LongSkywork: A Training Recipe for Efficiently Extending Context Length in Large Language Modelscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 6.

13 papers.

Sort:

By area:

  1. 2025Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routingcited by 13
  2. 2025Beyond Hard and Soft: Hybrid Context Compression for Balancing Local and Global Information Retentioncited by 7
  3. 2025FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernelcited by 7
  4. 2025Training Foundation Models on a Full-Stack AMD Platform: Compute, Networking, and System Designcited by 5
  5. 2026FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Servingcited by 5
  6. 2026VeriCache: Turning Lossy KV Cache into Lossless LLM Inferencecited by 5
  7. 2025Scaling External Knowledge Input Beyond Context Windows of LLMs via Multi-Agent Collaborationcited by 4
  8. 2026Out of the Memory Barrier: A Highly Memory Efficient Training System for LLMs with Million-Token Contextscited by 4
  9. 2025ILRe: Intermediate Layer Retrieval for Context Compression in Causal Language Modelscited by 3
  10. 2026OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modelingcited by 1
  11. 2025PaceLLM: Brain-Inspired Large Language Models for Long-Context Understandingcited by 0
  12. 2026Shuffle the Context: RoPE-Perturbed Self-Distillation for Long-Context Adaptationcited by 0
1 more
  1. 2026Beyond Memory: Harnessing Long-Horizon Agents with Explicit Belief Statescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Scaling Instruction-Tuned LLMs to Million-Token Contexts via Hierarchical Synthetic Data Generation
  2. 2024A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts
  3. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models
  4. 2025Generalizing From Short to Long: Effective Data Synthesis for Long-Context Instruction Tuning
  5. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Models
  6. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory
  7. 2024World Model on Million-Length Video And Language With Blockwise RingAttention
  8. 2025Long-Short Alignment for Effective Long-Context Modeling in LLMs