mlsanityData to 7 Oct 2026

NExtLong: Toward Effective Long-Context Training without Long Documents

arXiv 2501.12766 ↗First posted 22 Jan 2025cs.CL cs.AIcited by 19

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) with extended context windows have made significant strides yet remain a challenge due to the scarcity of long documents. Existing methods tend to synthesize long-context data but lack a clear mechanism to reinforce the long-range dependency modeling. To address this limitation, we propose NExtLong, a novel framework for synthesizing long-context data through Negative document Extension. NExtLong decomposes a document into multiple meta-chunks and extends the context by interleaving hard negative distractors retrieved from pretraining corpora. This approach compels the model to discriminate long-range dependent context from distracting content, enhancing its ability to model long-range dependencies. Extensive experiments demonstrate that NExtLong achieves significant performance improvements on the HELMET and RULER benchmarks compared to existing long-context synthesis approaches and leading models, which are trained on non-synthetic long documents. These findings highlight NExtLong's ability to reduce reliance on non-synthetic long documents, making it an effective framework for developing advanced long-context LLMs.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 67 of its 99 references that are in the map. First posted on arXiv 22 Jan 2025.

Builds on

67 of 99 references.

References · 67

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
38 more
  1. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  3. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  4. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  7. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  8. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  10. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  11. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  12. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  13. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  14. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  15. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  16. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  17. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  18. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  19. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  20. 2020Contrastive Learning with Hard Negative Samplescited by 619
  21. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  22. 2024Long Context Transfer from Language to Visioncited by 502
  23. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  24. 2020Hard Negative Mixing for Contrastive Learningcited by 399
  25. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  26. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  27. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  28. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  29. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  30. 2019Document Expansion by Query Predictioncited by 225
  31. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  32. 2024Many-Shot In-Context Learningcited by 203
  33. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  34. 2021Optimizing Dense Retrieval Model Training with Hard Negativescited by 154
  35. 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Textcited by 151
  36. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  37. 2023CodePlan: Repository-level Coding using LLMs and Planningcited by 139
  38. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 11.

19 papers.

Sort:

By area:

  1. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  2. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 235
  3. 2025LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructionscited by 6
  4. 2025LiteraryQA: Towards Effective Evaluation of Long-document Narrative QAcited by 6
  5. 2025EntropyLong: Effective Long-Context Training via Predictive Uncertaintycited by 5
  6. 2025Revisiting Long-context Modeling from Context Denoising Perspectivecited by 5
  7. 2025LiteLong: Resource-Efficient Long-Context Data Synthesis for LLMscited by 3
  8. 2025Libra: Large Chinese-based Safeguard for AI Contentcited by 2
  9. 2026How to Train Your Long-Context Visual Document Modelcited by 2
  10. 2026DySCO: Dynamic Attention-Scaling Decoding for Long-Context Language Modelscited by 2
  11. 2026Incentivizing In-depth Reasoning over Long Contexts with Process Advantage Shapingcited by 1
  12. 2026Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RLcited by 1
7 more
  1. 2026OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modelingcited by 1
  2. 2026Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledgecited by 1
  3. 2026Internalized Reasoning for Long-Context Visual Document Understandingcited by 0
  4. 2026PolicyLong: Towards On-Policy Context Extensioncited by 0
  5. 2026XekRung Technical Reportcited by 0
  6. 2026The Announcement Carries the Cue: Markup, Boundaries, and the Notation of Pre-Training Corporacited by 0
  7. 2026Doc-REFRAG: Rethinking Multimodal Document Retrieval-Augmented Generationcited by 0