mlsanityData to 7 Oct 2026

Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor

arXiv 2212.09689 ↗First posted 19 Dec 2022cs.CL cs.AI cs.LGcited by 395

Not on the map. Why?

Cited by 37 in Synthetic data and self-improvement for LLMs, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Instruction tuning enables pretrained language models to perform new tasks from inference-time natural language descriptions. These approaches rely on vast amounts of human supervision in the form of crowdsourced datasets or user interactions. In this work, we introduce Unnatural Instructions: a large dataset of creative and diverse instructions, collected with virtually no human labor. We collect 64,000 examples by prompting a language model with three seed examples of instructions and eliciting a fourth. This set is then expanded by prompting the model to rephrase each instruction, creating a total of approximately 240,000 examples of instructions, inputs, and outputs. Experiments show that despite containing a fair amount of noise, training on Unnatural Instructions rivals the effectiveness of training on open-source manually-curated datasets, surpassing the performance of models such as T0++ and Tk-Instruct across various benchmarks. These results demonstrate the potential of model-generated data as a cost-effective alternative to crowdsourcing for dataset expansion and diversification.

Key facts. Cited by 395 papers in the map (data through 7 Oct 2026). Builds on 35 of its 44 references that are in the map. First posted on arXiv 19 Dec 2022.

Builds on

35 of 44 references.

References · 35

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  9. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  10. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  11. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  12. 2015A large annotated corpus for learning natural language inferencecited by 1,837
23 more
  1. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  2. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  3. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  4. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  5. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  6. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  7. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  8. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  9. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  10. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
  11. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  12. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
  13. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 209
  14. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 200
  15. 2021Generating Datasets with Pretrained Language Modelscited by 192
  16. 2019Not Enough Data? Deep Learning to the Rescue!cited by 174
  17. 2019Good-Enough Compositional Data Augmentationcited by 158
  18. 2020Few-Shot Text Generation with Pattern-Exploiting Trainingcited by 87
  19. 2020The Turking Test: Can Language Models Understand Instructions?cited by 82
  20. 2021Neural Data Augmentation via Example Extrapolationcited by 54
  21. 2021Reordering Examples Helps during Priming-based Few-Shot Learningcited by 50
  22. 2022QAmeleon: Multilingual QA with Only 5 Examplescited by 25
  23. 2022LMentry: A Language Model Benchmark of Elementary Language Taskscited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 122; 2024: 160; 2025: 68; 2026: 44.

395 papers.

Sort:

By area:

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  6. 2023A Survey of Large Language Modelscited by 2,537
  7. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  8. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  9. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  10. 2023LIMA: Less Is More for Alignmentcited by 1,080
  11. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  12. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
38 more
  1. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  2. 2023Instruction Tuning with GPT-4cited by 676
  3. 2024Self-Rewarding Language Modelscited by 654
  4. 2023A Comprehensive Overview of Large Language Modelscited by 639
  5. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  6. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  7. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  8. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  9. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  10. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  11. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 383
  12. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  13. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 301
  14. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 299
  15. 2023Improving Text Embeddings with Large Language Modelscited by 295
  16. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  17. 2023The False Promise of Imitating Proprietary LLMscited by 254
  18. 2023Challenges and Applications of Large Language Modelscited by 251
  19. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 251
  20. 2023FireAct: Toward Language Agent Fine-tuningcited by 247
  21. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 240
  22. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 200
  23. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  24. 2023Knowledge Editing for Large Language Models: A Surveycited by 189
  25. 2024CodeS: Towards Building Open-source Language Models for Text-to-SQLcited by 183
  26. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  27. 2023RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Modelscited by 173
  28. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  29. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 160
  30. 2023Self-Alignment with Instruction Backtranslationcited by 149
  31. 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 145
  32. 2023TableLlama: Towards Open Large Generalist Models for Tablescited by 136
  33. 2024Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuningcited by 134
  34. 2024AutoWebGLM: A Large Language Model-based Web Navigating Agentcited by 130
  35. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 129
  36. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questionscited by 125
  37. 2023#InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Modelscited by 108
  38. 2023MoDS: Model-oriented Data Selection for Instruction Tuningcited by 107

50 most cited of 395 All 395 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025From Real to Synthetic: Synthesizing Millions of Diversified and Complicated User Instructions with Attributed Grounding
  2. 2025Aligning Instruction Tuning with Pre-training
  3. 2024MAmmoTH2: Scaling Instructions from the Web
  4. 2024Unveiling Imitation Learning: Exploring the Impact of Data Falsity to Large Language Model
  5. 2025Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Models
  6. 2025BARE: Leveraging Base Language Models for Few-Shot Synthetic Data Generation
  7. 2023WizardLM: Empowering large pre-trained language models to follow complex instructions
  8. 2024Seed-Free Synthetic Data Generation Framework for Instruction-Tuning LLMs: A Case Study in Thai