mlsanityData to 14 Oct 2026

Instruction Induction: From Few Examples to Natural Language Task Descriptions

arXiv 2205.10782 ↗First posted 22 May 2022cs.CLcited by 152

Not on the map. Why?

Abstract (arXiv)

Large language models are able to perform a task by conditioning on a few input-output demonstrations - a paradigm known as in-context learning. We show that language models can explicitly infer an underlying task from a few demonstrations by prompting them to generate a natural language instruction that fits the examples. To explore this ability, we introduce the instruction induction challenge, compile a dataset consisting of 24 tasks, and define a novel evaluation metric based on executing the generated instruction. We discover that, to a large extent, the ability to generate instructions does indeed emerge when using a model that is both large enough and aligned to follow instructions; InstructGPT achieves 65.7% of human performance in our execution-based metric, while the original GPT-3 model reaches only 9.8% of human performance. This surprising result suggests that instruction induction might be a viable learning paradigm in and of itself, where instead of fitting a set of latent continuous parameters to the data, one searches for the best description in the natural language hypothesis space.

Key facts. Cited by 152 papers in the map (data through 14 Oct 2026). Builds on 39 of its 49 references that are in the map. First posted on arXiv 22 May 2022.

Builds on

39 of 49 references.

References · 39

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  10. 2016Pointer Sentinel Mixture Modelscited by 3,096
  11. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  12. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
27 more
  1. 2019Language Models as Knowledge Bases?cited by 1,936
  2. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  3. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  4. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  5. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  6. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  7. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  8. 2018Neural Network Acceptability Judgmentscited by 958
  9. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  10. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  11. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  12. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  13. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  14. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  15. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 606
  16. 2021MetaICL: Learning to Learn In Contextcited by 454
  17. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 372
  18. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 326
  19. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 243
  20. 2022Predictability and Surprise in Large Generative Modelscited by 210
  21. 2021Reframing Instructional Prompts to GPTk's Languagecited by 176
  22. 2021Meta-learning via Language Model In-context Tuningcited by 164
  23. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  24. 2022GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Modelscited by 127
  25. 2020The Turking Test: Can Language Models Understand Instructions?cited by 82
  26. 2022Describing Differences between Text Distributions with Natural Languagecited by 54
  27. 2017The RepEval 2017 Shared Task: Multi-Genre Natural Language Inference with Sentence Representationscited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 7; 2023: 56; 2024: 34; 2025: 32; 2026: 23.

152 papers.

Sort:

By area:

  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  2. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  3. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  4. 2022A Survey on In-context Learningcited by 721
  5. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 653
  6. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 551
  7. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 431
  8. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 310
  9. 2023Function Vectors in Large Language Modelscited by 308
  10. 2022Large Language Models are Few-Shot Clinical Information Extractorscited by 207
  11. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 201
  12. 2023Efficient Large Language Models: A Surveycited by 186
38 more
  1. 2023Large Language Models Understand and Can be Enhanced by Emotional Stimulicited by 146
  2. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 118
  3. 2023Decomposing Uncertainty for Large Language Models through Input Clarification Ensemblingcited by 114
  4. 2023Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Modelscited by 110
  5. 2023WebGLM: Towards An Efficient Web-Enhanced Question Answering System with Human Preferencescited by 100
  6. 2024LLaMEA: A Large Language Model Evolutionary Algorithm for Automatically Generating Metaheuristicscited by 97
  7. 2023Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinementcited by 89
  8. 2023Large Language Models and Knowledge Graphs: Opportunities and Challengescited by 71
  9. 2024Hypothesis Generation with Large Language Modelscited by 71
  10. 2024A Systematic Survey on Large Language Models for Algorithm Designcited by 69
  11. 2023InstructZero: Efficient Instruction Optimization for Black-Box Large Language Modelscited by 68
  12. 2025A Systematic Survey of Automatic Prompt Optimization Techniquescited by 63
  13. 2023Goal Driven Discovery of Distributional Differences via Language Descriptionscited by 62
  14. 2023Dynamic Planning with a LLMcited by 61
  15. 2023Prompt Engineering a Prompt Engineercited by 53
  16. 2024Universal Vulnerabilities in Large Language Models: Backdoor Attacks for In-context Learningcited by 53
  17. 2024Synatra: Turning Indirect Knowledge into Direct Demonstrations for Digital Agents at Scalecited by 43
  18. 2023Dynosaur: A Dynamic Growth Paradigm for Instruction-Tuning Data Curationcited by 38
  19. 2024Efficient Prompt Optimization Through the Lens of Best Arm Identificationcited by 37
  20. 2024GoldCoin: Grounding Large Language Models in Privacy Laws via Contextual Integrity Theorycited by 37
  21. 2025Emergent Abilities in Large Language Models: A Surveycited by 37
  22. 2023Language Models as Black-Box Optimizers for Vision-Language Modelscited by 36
  23. 2022Robustness of Learning from Task Instructionscited by 35
  24. 2024The broader spectrum of in-context learningcited by 35
  25. 2024Evaluating Language Models as Synthetic Data Generatorscited by 33
  26. 2023Joint Prompt Optimization of Stacked LLMs using Variational Inferencecited by 32
  27. 2023Large Language Model Instruction Following: A Survey of Progresses and Challengescited by 31
  28. 2023AutoHint: Automatic Prompt Optimization with Hint Generationcited by 30
  29. 2023Instruct Me More! Random Prompting for Visual In-Context Learningcited by 30
  30. 2023InstructProtein: Aligning Human and Protein Language via Knowledge Instructioncited by 28
  31. 2023On the Detectability of ChatGPT Content: Benchmarking, Methodology, and Evaluation through the Lens of Academic Writingcited by 27
  32. 2023Safer-Instruct: Aligning Language Models with Automated Preference Datacited by 27
  33. 2023Layout and Task Aware Instruction Prompt for Zero-shot Document Image Question Answeringcited by 24
  34. 2023Guideline Learning for In-context Information Extractioncited by 24
  35. 2024Localized Zeroth-Order Prompt Optimizationcited by 23
  36. 2022Guess the Instruction! Flipped Learning Makes Language Models Stronger Zero-Shot Learnerscited by 22
  37. 2023Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Taskscited by 22
  38. 2023Instruction Tuned Models are Quick Learnerscited by 21

50 most cited of 152 All 152 (JSON)