mlsanityData to 7 Oct 2026

Instruction Pre-Training: Language Models are Supervised Multitask Learners

arXiv 2406.14491 ↗First posted 20 Jun 2024cs.CLcited by 63

Not on the map. Why?

Abstract (arXiv)

Unsupervised multitask pre-training has been the critical method behind the recent success of language models (LMs). However, supervised multitask learning still holds significant promise, as scaling it in the post-training stage trends towards better generalization. In this paper, we explore supervised multitask pre-training by proposing Instruction Pre-Training, a framework that scalably augments massive raw corpora with instruction-response pairs to pre-train LMs. The instruction-response pairs are generated by an efficient instruction synthesizer built on open-source models. In our experiments, we synthesize 200M instruction-response pairs covering 40+ task categories to verify the effectiveness of Instruction Pre-Training. In pre-training from scratch, Instruction Pre-Training not only consistently enhances pre-trained base models but also benefits more from further instruction tuning. In continual pre-training, Instruction Pre-Training enables Llama3-8B to be comparable to or even outperform Llama3-70B. Our model, code, and data are available at https://github.com/microsoft/LMOps.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Builds on 79 of its 103 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

79 of 103 references.

References · 79

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  11. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
38 more
  1. 2023Qwen Technical Reportcited by 3,273
  2. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  3. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  6. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  7. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  10. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  11. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  12. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  13. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  14. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  15. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  16. 2023LIMA: Less Is More for Alignmentcited by 1,080
  17. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  18. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  19. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  20. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  21. 2018CoQA: A Conversational Question Answering Challengecited by 674
  22. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  23. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  24. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  25. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  26. 2023The Falcon Series of Open Language Modelscited by 526
  27. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  28. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  29. 2021MetaICL: Learning to Learn In Contextcited by 450
  30. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  31. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  32. 2018QuAC : Question Answering in Contextcited by 379
  33. 2016FastText.zip: Compressing text classification modelscited by 331
  34. 2016NewsQA: A Machine Comprehension Datasetcited by 330
  35. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
  36. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 309
  37. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 281
  38. 2019QASC: A Dataset for Question Answering via Sentence Compositioncited by 248

50 most cited of 79 All 79 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 26; 2026: 21.

63 papers.

Sort:

By area:

  1. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 153
  2. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 127
  3. 2025Time-MQA: Time Series Multi-Task Question Answering with Context Enhancementcited by 74
  4. 2024Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMscited by 39
  5. 2024Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scalecited by 37
  6. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Modelscited by 29
  7. 2024SciLitLLM: How to Adapt LLMs for Scientific Literature Understandingcited by 29
  8. 2024SimRAG: Self-Improving Retrieval-Augmented Generation for Adapting Large Language Models to Specialized Domainscited by 28
  9. 2024MiniPLM: Knowledge Distillation for Pre-Training Language Modelscited by 27
  10. 2024Unlocking Reasoning Potential in Large Langauge Models by Scaling Code-form Planningcited by 24
  11. 2024How to Synthesize Text Data without Model Collapse?cited by 24
  12. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfallscited by 24
38 more
  1. 2025A Survey on LLM Mid-Trainingcited by 18
  2. 2024Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Expertscited by 17
  3. 2025QuaDMix: Quality-Diversity Balanced Data Selection for Efficient LLM Pretrainingcited by 16
  4. 2024Mixing It Up: The Cocktail Effect of Multi-Task Fine-Tuning on LLM Performance -- A Case Study in Financecited by 15
  5. 2026Graph-of-Agents: A Graph-based Framework for Multi-Agent LLM Collaborationcited by 15
  6. 2024On Domain-Adaptive Post-Training for Multimodal Large Language Modelscited by 13
  7. 2025Domaino1s: Guiding LLM Reasoning for Explainable Answers in High-Stakes Domainscited by 11
  8. 2026Computer Environments Elicit General Agentic Intelligence in LLMscited by 11
  9. 2025DCR: Quantifying Data Contamination in LLMs Evaluationcited by 10
  10. 2025Midtraining Bridges Pretraining and Posttraining Distributionscited by 10
  11. 2025SearchRAG: Can Search Engines Be Helpful for LLM-based Medical Question Answering?cited by 9
  12. 2024Semi-supervised Fine-tuning for Large Language Modelscited by 8
  13. 2025Does Learning Mathematical Problem-Solving Generalize to Broader Reasoning?cited by 7
  14. 2026Agent Lightning v1.0: Towards Harnessed Agentic RLcited by 7
  15. 2024Combining Domain and Alignment Vectors to Achieve Better Knowledge-Safety Trade-offs in LLMscited by 6
  16. 2025CoddLLM: Empowering Large Language Models for Data Analyticscited by 6
  17. 2026How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Datacited by 6
  18. 2025A Survey of LLM × DATAcited by 4
  19. 2026Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Taskscited by 4
  20. 2026Beyond Static Models: An Evolving Framework for Continual Learning in Large Language Models across Training Stagescited by 4
  21. 2024Mastering the Craft of Data Synthesis for CodeLLMscited by 3
  22. 2025Localizing Persona Representations in LLMscited by 3
  23. 2025Large-Scale Diverse Synthesis for Mid-Trainingcited by 3
  24. 2025Select to Know: An Internal-External Knowledge Self-Selection Framework for Domain-Specific Question Answeringcited by 3
  25. 2025Towards EnergyGPT: A Large Language Model Specialized for the Energy Sectorcited by 3
  26. 2024OnlySportsLM: Optimizing Sports-Domain Language Models with SOTA Performance under Billion Parameterscited by 2
  27. 2025DocTalk: Scalable Graph-based Dialogue Synthesis for Enhancing LLM Conversational Capabilitiescited by 2
  28. 2025LinkQA: Synthesizing Diverse QA from Multiple Seeds Strongly Linked by Knowledge Pointscited by 2
  29. 2025Comparing Knowledge Injection Methods for LLMs in a Low-Resource Regimecited by 2
  30. 2025From Amateur to Master: Infusing Knowledge into LLMs via Automated Curriculum Learningcited by 2
  31. 2025Merging Continual Pretraining Models for Domain-Specialized LLMs: A Case Study in Financecited by 2
  32. 2026ExpGuard: LLM Content Moderation in Specialized Domainscited by 2
  33. 2025Understanding Fact Recall in Language Models: Why Two-Stage Training Encourages Memorization but Mixed Training Teaches Knowledgecited by 1
  34. 2025DACIP-RC: Domain Adaptive Continual Instruction Pre-Training via Reading Comprehension on Business Conversationscited by 1
  35. 2025Token-Level Marginalization for Multi-Label LLM Classifierscited by 1
  36. 2026EstLLM: Enhancing Estonian Capabilities in Multilingual LLMs via Continued Pretraining and Post-Trainingcited by 1
  37. 2026MultiHashFormer: Hash-based Generative Language Modelscited by 1
  38. 2025Using Scaling Laws for Data Source Utility Estimation in Domain-Specific Pre-Trainingcited by 0

50 most cited of 63 All 63 (JSON)