mlsanityData to 14 Oct 2026

LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions

arXiv 2505.17134 ↗First posted 22 May 2025cs.CL cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

High-quality long-context instruction data is essential for aligning long-context large language models (LLMs). Despite the public release of models like Qwen and Llama, their long-context instruction data remains proprietary. Human annotation is costly and challenging, while template-based synthesis methods limit scale, diversity, and quality. We introduce LongMagpie, a self-synthesis framework that automatically generates large-scale long-context instruction data. Our key insight is that aligned long-context LLMs, when presented with a document followed by special tokens preceding a user turn, auto-regressively generate contextually relevant queries. By harvesting these document-query pairs and the model's responses, LongMagpie produces high-quality instructions without human effort. Experiments on HELMET, RULER, and Longbench v2 demonstrate that LongMagpie achieves leading performance on long-context tasks while maintaining competitive performance on short-context tasks, establishing it as a simple and effective approach for open, diverse, and scalable long-context instruction data synthesis.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 49 of its 72 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

49 of 72 references.

References · 49

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  11. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  12. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
37 more
  1. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  2. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  3. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  4. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  5. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  6. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  7. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  8. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  9. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  10. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  11. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  12. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
  13. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  14. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  15. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  16. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 267
  17. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 244
  18. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 232
  19. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 186
  20. 2023Self-Alignment with Instruction Backtranslationcited by 151
  21. 2024How to Train Long-Context Language Models (Effectively)cited by 146
  22. 2024jina-embeddings-v3: Multilingual Embeddings With Task LoRAcited by 141
  23. 2023CodePlan: Repository-level Coding using LLMs and Planningcited by 140
  24. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 111
  25. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Modelscited by 106
  26. 2024Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Modelscited by 99
  27. 2023In-context Pretraining: Language Modeling Beyond Document Boundariescited by 79
  28. 2024ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilitiescited by 53
  29. 2023Peek Across: Improving Multi-Document Modeling via Cross-Document Question-Answeringcited by 28
  30. 2023Less is More for Long Document Summary Evaluation by LLMscited by 28
  31. 2024GATEAU: Selecting Influential Samples for Long Context Alignmentcited by 20
  32. 2025NExtLong: Toward Effective Long-Context Training without Long Documentscited by 20
  33. 2024Quest: Query-centric Data Synthesis Approach for Long-context Scaling of Large Language Modelcited by 16
  34. 2024LOGO -- Long cOntext aliGnment via efficient preference Optimizationcited by 9
  35. 2025WildLong: Synthesizing Realistic Long-Context Instruction Data at Scalecited by 9
  36. 2025Generalizing From Short to Long: Effective Data Synthesis for Long-Context Instruction Tuningcited by 8
  37. 2024Bootstrap Your Own Context Lengthcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2026C2KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inferencecited by 4
  2. 2025Libra: Large Chinese-based Safeguard for AI Contentcited by 2
  3. 2026OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modelingcited by 2
  4. 2026PolicyLong: Towards On-Policy Context Extensioncited by 0
  5. 2026LongCrafter: Towards Diverse Long-Context Understanding via Evidence-Graph-Guided Instruction Synthesiscited by 0
  6. 2026Unsupervised Post-Training of Foundation Models: A Surveycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Scaling Instruction-Tuned LLMs to Million-Token Contexts via Hierarchical Synthetic Data Generation
  2. 2025From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models
  3. 2025LongProc: Benchmarking Long-Context Language Models on Long Procedural Generation
  4. 2024CodecLM: Aligning Language Models with Tailored Synthetic Data
  5. 2025From Real to Synthetic: Synthesizing Millions of Diversified and Complicated User Instructions with Attributed Grounding
  6. 2024Benchmarking Large Language Models on Controllable Generation under Diversified Instructions
  7. 2025Flora: Effortless Context Construction to Arbitrary Length and Scale
  8. 2024MAmmoTH2: Scaling Instructions from the Web