mlsanityData to 7 Oct 2026

WildLong: Synthesizing Realistic Long-Context Instruction Data at Scale

arXiv 2502.16684 ↗First posted 23 Feb 2025cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) with extended context windows enable tasks requiring extensive information integration but are limited by the scarcity of high-quality, diverse datasets for long-context instruction tuning. Existing data synthesis methods focus narrowly on objectives like fact retrieval and summarization, restricting their generalizability to complex, real-world tasks. WildLong extracts meta-information from real user queries, models co-occurrence relationships via graph-based methods, and employs adaptive generation to produce scalable data. It extends beyond single-document tasks to support multi-document reasoning, such as cross-document comparison and aggregation. Our models, finetuned on 150K instruction-response pairs synthesized using WildLong, surpasses existing open-source long-context-optimized models across benchmarks while maintaining strong performance on short-context tasks without incorporating supplementary short-context data. By generating a more diverse and realistic long-context instruction dataset, WildLong enhances LLMs' ability to generalize to complex, real-world reasoning over long contexts, establishing a new paradigm for long-context data synthesis.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 67 of its 78 references that are in the map. First posted on arXiv 23 Feb 2025.

Builds on

67 of 78 references.

References · 67

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2024Qwen2.5 Technical Reportcited by 4,713
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  10. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  11. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  12. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
38 more
  1. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  2. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  4. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  5. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  6. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  7. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  8. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  9. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  10. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  11. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  12. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  13. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  14. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  15. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 397
  16. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  17. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  18. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  19. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
  20. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  21. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  22. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 201
  23. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  24. 2023Augmenting Language Models with Long-Term Memorycited by 186
  25. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  26. 2024Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Modelscited by 150
  27. 2024MathScale: Scaling Instruction Tuning for Mathematical Reasoningcited by 144
  28. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  29. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 133
  30. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 127
  31. 2023Scaling Laws of RoPE-based Extrapolationcited by 124
  32. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QAcited by 121
  33. 2024Make Your LLM Fully Utilize the Contextcited by 118
  34. 2023Retrieval meets Long Context Large Language Modelscited by 114
  35. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 109
  36. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Modelscited by 102
  37. 2024Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Modelscited by 99
  38. 2024Stable LM 2 1.6B Technical Reportcited by 86

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 6.

9 papers.

  1. 2025LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructionscited by 6
  2. 2025AcademicEval: Live Long-Context LLM Benchmarkcited by 4
  3. 2026How to Train Your Long-Context Visual Document Modelcited by 2
  4. 2025Modular Techniques for Synthetic Long-Context Data Generation in Language Model Training and Evaluationcited by 1
  5. 2026Incentivizing In-depth Reasoning over Long Contexts with Process Advantage Shapingcited by 1
  6. 2026Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RLcited by 1
  7. 2026OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modelingcited by 1
  8. 2026Internalized Reasoning for Long-Context Visual Document Understandingcited by 0
  9. 2026LongCrafter: Towards Diverse Long-Context Understanding via Evidence-Graph-Guided Instruction Synthesiscited by 0