mlsanityData to 7 Oct 2026

Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing

arXiv 2406.08464 ↗First posted 12 Jun 2024cs.CL cs.AIcited by 341

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 2 more
Abstract (arXiv)

High-quality instruction data is critical for aligning large language models (LLMs). Although some models, such as Llama-3-Instruct, have open weights, their alignment data remain private, which hinders the democratization of AI. High human labor costs and a limited, predefined scope for prompting prevent existing open-source data creation methods from scaling effectively, potentially limiting the diversity and quality of public alignment datasets. Is it possible to synthesize high-quality instruction data at scale by extracting it directly from an aligned LLM? We present a self-synthesis method for generating large-scale alignment data named Magpie. Our key observation is that aligned LLMs like Llama-3-Instruct can generate a user query when we input only the left-side templates up to the position reserved for user messages, thanks to their auto-regressive nature. We use this method to prompt Llama-3-Instruct and generate 4 million instructions along with their corresponding responses. We perform a comprehensive analysis of the extracted data and select 300K high-quality instances. To compare Magpie data with other public instruction datasets, we fine-tune Llama-3-8B-Base with each dataset and evaluate the performance of the fine-tuned models. Our results indicate that in some tasks, models fine-tuned with Magpie perform comparably to the official Llama-3-8B-Instruct, despite the latter being enhanced with 10 million data points through supervised fine-tuning (SFT) and subsequent feedback learning. We also show that using Magpie solely for SFT can surpass the performance of previous public datasets utilized for both SFT and preference optimization, such as direct preference optimization with UltraFeedback. This advantage is evident on alignment benchmarks such as AlpacaEval, ArenaHard, and WildBench.

Key facts. Cited by 341 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 52 of its 86 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

52 of 86 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning4 Dec
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May

Other references · 50

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  12. 2023Qwen Technical Reportcited by 3,273
38 more
  1. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  3. 2020Extracting Training Data from Large Language Modelscited by 2,144
  4. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  5. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  6. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  7. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  8. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  9. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  10. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  11. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  12. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  13. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  14. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  15. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  16. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  17. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  18. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  19. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  20. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 354
  21. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  22. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  23. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 280
  24. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227
  25. 2022What Does it Mean for a Language Model to Preserve Privacy?cited by 194
  26. 2024Are We Done with MMLU?cited by 178
  27. 2023Emergent and Predictable Memorization in Large Language Modelscited by 147
  28. 2019Thieves on Sesame Street! Model Extraction of BERT-based APIscited by 136
  29. 2024Best Practices and Lessons Learned on Synthetic Datacited by 104
  30. 2024Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Modelscited by 99
  31. 2023Bag of Tricks for Training Data Extraction from Language Modelscited by 67
  32. 2022On the Limitations of Reference-Free Evaluations of Generated Textcited by 52
  33. 2023WaveCoder: Widespread And Versatile Enhancement For Code Large Language Models By Instruction Tuningcited by 42
  34. 2024CodecLM: Aligning Language Models with Tailored Synthetic Datacited by 39
  35. 2023Dynosaur: A Dynamic Growth Paradigm for Instruction-Tuning Data Curationcited by 38
  36. 2024Better Synthetic Data by Retrieving and Transforming Existing Datasetscited by 36
  37. 2024Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMscited by 33
  38. 2024Special Characters Attack: Toward Scalable Training Data Extraction From Large Language Modelscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 61; 2025: 173; 2026: 107.

341 papers.

Sort:

By area:

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  3. 2025Large Language Diffusion Modelscited by 872
  4. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  5. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  6. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  7. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 372
  8. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 322
  9. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  10. 2025Kimi-Audio Technical Reportcited by 210
  11. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  12. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 118
38 more
  1. 2025KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Codingcited by 115
  2. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 115
  3. 2025Small Models Struggle to Learn from Strong Reasonerscited by 109
  4. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  5. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languagescited by 71
  6. 2024Reinforcement Learning Enhanced LLMs: A Surveycited by 69
  7. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  8. 2024FuseChat: Knowledge Fusion of Chat Modelscited by 53
  9. 2025ILLUME+: Illuminating Unified MLLM with Dual Visual Tokenization and Diffusion Refinementcited by 53
  10. 2024On Evaluating the Durability of Safeguards for Open-Weight LLMscited by 51
  11. 2025Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Modelcited by 50
  12. 2025Infinity Instruct: Scaling Instruction Selection and Synthesis to Enhance Language Modelscited by 49
  13. 2025The Best Instruction-Tuning Data are Those That Fitcited by 48
  14. 2025FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocitiescited by 46
  15. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 45
  16. 2024EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotionscited by 44
  17. 2025Scaling Laws of Synthetic Data for Language Modelscited by 44
  18. 2025Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representationscited by 44
  19. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignmentcited by 42
  20. 2024A Survey on Data Synthesis and Augmentation for Large Language Modelscited by 40
  21. 2025Bring Reason to Vision: Understanding Perception and Reasoning through Model Mergingcited by 40
  22. 2025Tower+: Bridging Generality and Translation Specialization in Multilingual LLMscited by 40
  23. 2025ViSpeak: Visual Instruction Feedback in Streaming Videoscited by 38
  24. 2024ScaleBiO: Scalable Bilevel Optimization for LLM Data Reweightingcited by 35
  25. 2024Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarkingcited by 35
  26. 2026On Data Engineering for Scaling LLM Terminal Capabilitiescited by 34
  27. 2024Evaluating Language Models as Synthetic Data Generatorscited by 32
  28. 2025MergeBench: A Benchmark for Merging Domain-Specialized LLMscited by 30
  29. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Modelscited by 29
  30. 2026SkillGen: Verified Inference-Time Agent Skill Synthesiscited by 29
  31. 2025PARD: Accelerating LLM Inference with Low-Cost PARallel Draft Model Adaptationcited by 28
  32. 2024I-SHEEP: Self-Alignment of LLM from Scratch through an Iterative Self-Enhancement Paradigmcited by 27
  33. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  34. 2025OmniAlign-V: Towards Enhanced Alignment of MLLMs with Human Preferencecited by 27
  35. 2025Building a Foundational Guardrail for General Agentic Systems via Synthetic Datacited by 26
  36. 2024LLaMA-MoE v2: Exploring Sparsity of LLaMA from Perspective of Mixture-of-Experts with Post-Trainingcited by 25
  37. 2024Unleashing LLM Reasoning Capability via Scalable Question Synthesis from Scratchcited by 24
  38. 2025Genius: A Generalizable and Purely Unsupervised Self-Training Framework For Advanced Reasoningcited by 24

50 most cited of 341 All 341 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Efficient Alignment of Large Language Models via Data Sampling
  2. 2024Align2LLaVA: Cascaded Human and Large Language Model Preference Alignment for Multi-modal Instruction Curation
  3. 2026ECLIPTICA -- A Framework for Switchable LLM Alignment via CITA - Contrastive Instruction-Tuned Alignment
  4. 2024Human-Instruction-Free LLM Self-Alignment with Limited Samples
  5. 2023Aligner: One Global Token is Worth Millions of Parameters When Aligning Large Language Models
  6. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Models
  7. 2025MAIN: Mutual Alignment Is Necessary for instruction tuning
  8. 2024Is Free Self-Alignment Possible?