mlsanityData to 7 Oct 2026

Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models

arXiv 2402.13064 ↗First posted 20 Feb 2024cs.CLcited by 99

On the map: Synthetic data and self-improvement for LLMs → Code and instruction data synthesis

Abstract (arXiv)

We introduce Generalized Instruction Tuning (called GLAN), a general and scalable method for instruction tuning of Large Language Models (LLMs). Unlike prior work that relies on seed examples or existing datasets to construct instruction tuning data, GLAN exclusively utilizes a pre-curated taxonomy of human knowledge and capabilities as input and generates large-scale synthetic instruction data across all disciplines. Specifically, inspired by the systematic structure in human education system, we build the taxonomy by decomposing human knowledge and capabilities to various fields, sub-fields and ultimately, distinct disciplines semi-automatically, facilitated by LLMs. Subsequently, we generate a comprehensive list of subjects for every discipline and proceed to design a syllabus tailored to each subject, again utilizing LLMs. With the fine-grained key concepts detailed in every class session of the syllabus, we are able to generate diverse instructions with a broad coverage across the entire spectrum of human knowledge and skills. Extensive experiments on large language models (e.g., Mistral) demonstrate that GLAN excels in multiple dimensions from mathematical reasoning, coding, academic exams, logical reasoning to general instruction following without using task-specific training data of these tasks. In addition, GLAN allows for easy customization and new fields or skills can be added by simply incorporating a new node into our taxonomy.

Key facts. Cited by 99 papers in the map (data through 7 Oct 2026). Placed in Code and instruction data synthesis, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 33 of its 47 references that are in the map. First posted on arXiv 20 Feb 2024.

Builds on

33 of 47 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec

Other references · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2020Scaling Laws for Neural Language Modelscited by 6,411
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
20 more
  1. 2021Program Synthesis with Large Language Modelscited by 3,509
  2. 2023Mistral 7Bcited by 3,065
  3. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  5. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  6. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  7. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  8. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  9. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  10. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  11. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  12. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  13. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  14. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  15. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  16. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  17. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  18. 2023The False Promise of Imitating Proprietary LLMscited by 254
  19. 2023Large Language Model Alignment: A Surveycited by 241
  20. 2023Orca 2: Teaching Small Language Models How to Reasoncited by 157

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 35; 2025: 47; 2026: 17.

99 papers.

Sort:

By area:

  1. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 397
  2. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  3. 2024xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Tokencited by 123
  4. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  5. 2024Instruction Pre-Training: Language Models are Supervised Multitask Learnerscited by 63
  6. 2024Key-Point-Driven Data Synthesis with its Enhancement on Mathematical Reasoningcited by 58
  7. 2024Synthetic continued pretrainingcited by 56
  8. 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Modelscited by 56
  9. 2024Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Taskscited by 52
  10. 2025SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Datacited by 50
  11. 2025mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Datacited by 49
  12. 2024JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Modelscited by 44
38 more
  1. 2025Scaling Laws of Synthetic Data for Language Modelscited by 44
  2. 2025ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMscited by 44
  3. 2024Self-Boosting Large Language Models with Synthetic Preference Datacited by 40
  4. 2024A Survey on Data Synthesis and Augmentation for Large Language Modelscited by 40
  5. 2024Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMscited by 39
  6. 2025TIMER: Temporal Instruction Modeling and Evaluation for Longitudinal Clinical Recordscited by 33
  7. 2024Self-Pluralising Culture Alignment for Large Language Modelscited by 31
  8. 2025BitNet b1.58 2B4T Technical Reportcited by 30
  9. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Modelscited by 29
  10. 2026SkillGen: Verified Inference-Time Agent Skill Synthesiscited by 29
  11. 2025OpenCharacter: Training Customizable Role-Playing LLMs with Large-Scale Synthetic Personascited by 25
  12. 2024Unleashing LLM Reasoning Capability via Scalable Question Synthesis from Scratchcited by 24
  13. 2024Self-MoE: Towards Compositional Large Language Models with Self-Specialized Expertscited by 22
  14. 2025Data Whisperer: Efficient Data Selection for Task-Specific LLM Fine-Tuning via Few-Shot In-Context Learningcited by 19
  15. 2024Stronger Models are NOT Stronger Teachers for Instruction Tuningcited by 16
  16. 2024Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Modelscited by 16
  17. 2024LLM Roleplay: Simulating Human-Chatbot Interactioncited by 15
  18. 2025FSPO: Few-Shot Optimization of Synthetic Preferences Personalizes to Real Userscited by 14
  19. 2026A Survey on LLM-based Conversational User Simulationcited by 14
  20. 2024textitTrans-LoRA: towards data-free Transferable Parameter Efficient Finetuningcited by 13
  21. 2024Revisiting the Superficial Alignment Hypothesiscited by 12
  22. 2024KnowledgeSG: Privacy-Preserving Synthetic Text Generation with Knowledge Distillation from Servercited by 11
  23. 2024Little Giants: Synthesizing High-Quality Embedding Data at Scalecited by 11
  24. 2024Boosting Reward Model with Preference-Conditional Multi-Aspect Synthetic Data Generationcited by 9
  25. 2025WildLong: Synthesizing Realistic Long-Context Instruction Data at Scalecited by 9
  26. 2024Differential Transformercited by 8
  27. 2025BARE: Leveraging Base Language Models for Few-Shot Synthetic Data Generationcited by 8
  28. 2025Instruction-Tuning Data Synthesis from Scratch via Web Reconstructioncited by 8
  29. 2025Middo: Model-Informed Dynamic Data Optimization for Enhanced LLM Fine-Tuning via Closed-Loop Learningcited by 7
  30. 2025Self-Improving VLM Judges Without Human Annotationscited by 7
  31. 2024Task Oriented In-Domain Data Augmentationcited by 6
  32. 2024Multilingual Arbitrage: Optimizing Data Pools to Accelerate Multilingual Progresscited by 6
  33. 2025CrowdSelect: Synthetic Instruction Data Selection with Multi-LLM Wisdomcited by 6
  34. 2025TreeSynth: Synthesizing Diverse Data from Scratch via Tree-Guided Subspace Partitioningcited by 6
  35. 2025LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructionscited by 6
  36. 2025IFDECORATOR: Wrapping Instruction Following Reinforcement Learning with Verifiable Rewardscited by 6
  37. 2025ConsistentChat: Building Skeleton-Guided Consistent Multi-Turn Dialogues for Large Language Models from Scratchcited by 5
  38. 2024Value Alignment from Unstructured Textcited by 4

50 most cited of 99 All 99 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Models
  2. 2024MAmmoTH2: Scaling Instructions from the Web
  3. 2025Aligning Instruction Tuning with Pre-training
  4. 2026DS2-Instruct: Domain-Specific Data Synthesis for Large Language Models Instruction Tuning
  5. 2024Seed-Free Synthetic Data Generation Framework for Instruction-Tuning LLMs: A Case Study in Thai
  6. 2023Instruction Tuning with GPT-4
  7. 2023Self-Evolved Diverse Data Sampling for Efficient Instruction Tuning
  8. 2024Optimizing Instruction Synthesis: Effective Exploration of Evolutionary Space with Tree Search