mlsanityData to 14 Oct 2026

GenQA: Generating Millions of Instructions from a Handful of Prompts

arXiv 2406.10323 ↗First posted 14 Jun 2024cs.CLcited by 35

Not on the map. Why?

Abstract (arXiv)

Most public instruction finetuning datasets are relatively small compared to the closed source datasets used to train industry models. To study questions about finetuning at scale, such as curricula and learning rate cooldown schedules, there is a need for industrial-scale datasets. However, this scale necessitates a data generation process that is almost entirely automated. In this work, we study methods for generating large instruction datasets from a single prompt. With little human oversight, we get LLMs to write diverse sets of instruction examples ranging from simple completion tasks to complex multi-turn dialogs across a variety of subject areas. When finetuning a Llama-3 8B base model, our dataset meets or exceeds both WizardLM and Ultrachat on both knowledge-intensive leaderboard tasks as well as conversational evaluations. We release our dataset, the "generator" prompts that created it, and our finetuned model checkpoints.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Builds on 29 of its 39 references that are in the map. First posted on arXiv 14 Jun 2024.

Builds on

29 of 39 references.

References · 29

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  12. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
17 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  3. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  4. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  5. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  6. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  7. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  8. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  9. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  10. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  11. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 487
  12. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  13. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
  14. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 335
  15. 2022NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Taskscited by 112
  16. 2022ZeroPrompt: Scaling Prompt-Based Pretraining to 1,000 Tasks Improves Zero-Shot Generalizationcited by 54
  17. 2024Forcing Diffuse Distributions out of Language Modelscited by 48

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 22; 2026: 5.

35 papers.

Sort:

By area:

  1. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 122
  2. 2025The Best Instruction-Tuning Data are Those That Fitcited by 52
  3. 2025Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representationscited by 44
  4. 2025Training a Generally Curious Agentcited by 31
  5. 2025Zebra-Llama: Towards Extremely Efficient Hybrid Modelscited by 18
  6. 2024Stronger Models are NOT Stronger Teachers for Instruction Tuningcited by 16
  7. 2024Understanding Synthetic Context Extension via Retrieval Headscited by 10
  8. 2024MDCure: A Scalable Pipeline for Multi-Document Instruction-Followingcited by 10
  9. 2025MetaSynth: Meta-Prompting-Driven Agentic Scaffolds for Diverse Synthetic Data Generationcited by 10
  10. 2025ESC-Judge: A Framework for Comparing Emotional Support Conversational Agentscited by 8
  11. 2025RAD: Redundancy-Aware Distillation for Hybrid Models via Self-Speculative Decodingcited by 8
  12. 2025Improved Supervised Fine-Tuning for Large Language Models to Mitigate Catastrophic Forgettingcited by 8
23 more
  1. 2024GenAI Content Detection Task 2: AI vs. Human -- Academic Essay Authenticity Challengecited by 6
  2. 2025CrowdSelect: Synthetic Instruction Data Selection with Multi-LLM Wisdomcited by 6
  3. 2025From Real to Synthetic: Synthesizing Millions of Diversified and Complicated User Instructions with Attributed Groundingcited by 5
  4. 2024CSRec: Rethinking Sequential Recommendation from A Causal Perspectivecited by 4
  5. 2024Do Robot Snakes Dream like Electric Sheep? Investigating the Effects of Architectural Inductive Biases on Hallucinationcited by 4
  6. 2025X-EcoMLA: Upcycling Pre-Trained Attention into MLA for Efficient and Extreme KV Compressioncited by 4
  7. 2025TableDreamer: Progressive and Weakness-guided Data Synthesis from Scratch for Table Instruction Tuningcited by 4
  8. 2025Bohdi: Heterogeneous LLM Fusion with Automatic Data Explorationcited by 3
  9. 2025SpikingMamba: Towards Energy-Efficient Large Language Models via Knowledge Distillation from Mambacited by 3
  10. 2025Zero-Shot Vision Encoder Grafting via LLM Surrogatescited by 2
  11. 2026Tailoring the Curriculum: Student-Centered Reasoning Distillation via Dynamic Data-Model Compatibilitycited by 2
  12. 2024A Recipe For Building a Compliant Real Estate Chatbotcited by 1
  13. 2025FarSkip-Collective: Unhobbling Blocking Communication in Mixture of Experts Modelscited by 1
  14. 2025Auto-SPT: Automating Semantic Preserving Transformations for Codecited by 1
  15. 2026One Prompt, Many Sounds: Modeling Listener Variability in LLM-Based Equalizationcited by 1
  16. 2025ExpertGenQA: Open-ended QA generation in Specialized Domainscited by 0
  17. 2025FictionalQA: A Dataset for Studying Memorization and Knowledge Acquisitioncited by 0
  18. 2025jXBW: A Compressed Index Enabling Structure-Aware JSONL Retrieval for Structured RAGcited by 0
  19. 2025Match Chat: Real Time Generative AI and Generative Computing for Tenniscited by 0
  20. 2025How do Language Models Generate Slang: A Systematic Comparison between Human and Machine-Generated Slang Usagescited by 0
  21. 2026MAR: Efficient Large Language Models via Module-aware Architecture Refinementcited by 0
  22. 2026Multi-Token Residual Predictioncited by 0
  23. 2026Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMscited by 0