mlsanityData to 7 Oct 2026

Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks

arXiv 2204.07705 ↗First posted 16 Apr 2022cs.CL cs.AIcited by 858

Not on the map. Why?

Cited by 63 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

How well can NLP models generalize to a variety of unseen tasks when provided with task instructions? To address this question, we first introduce Super-NaturalInstructions, a benchmark of 1,616 diverse NLP tasks and their expert-written instructions. Our collection covers 76 distinct task types, including but not limited to classification, extraction, infilling, sequence tagging, text rewriting, and text composition. This large and diverse collection of tasks enables rigorous benchmarking of cross-task generalization under instructions -- training models to follow instructions on a subset of tasks and evaluating them on the remaining unseen ones. Furthermore, we build Tk-Instruct, a transformer model trained to follow a variety of in-context instructions (plain language task definitions or k-shot examples). Our experiments show that Tk-Instruct outperforms existing instruction-following models such as InstructGPT by over 9% on our benchmark despite being an order of magnitude smaller. We further analyze generalization as a function of various scaling parameters, such as the number of observed tasks, the number of instances per task, and model sizes. We hope our dataset and model facilitate future progress towards more general-purpose NLP models.

Key facts. Cited by 858 papers in the map (data through 7 Oct 2026). Builds on 38 of its 54 references that are in the map. First posted on arXiv 16 Apr 2022.

Builds on

38 of 54 references.

References · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  7. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  8. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  9. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  10. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  11. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  12. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
26 more
  1. 2021MetaICL: Learning to Learn In Contextcited by 450
  2. 2019Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examplescited by 312
  3. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  4. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 252
  5. 2022Can language models learn from explanations in context?cited by 248
  6. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 235
  7. 2022UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Modelscited by 231
  8. 2021Reframing Instructional Prompts to GPTk's Languagecited by 176
  9. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  10. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  11. 2020Language-Conditioned Imitation Learning for Robot Manipulation Taskscited by 136
  12. 2020Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehensioncited by 131
  13. 2021Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collectionscited by 128
  14. 2021FILM: Following Instructions in Language with Modular Methodscited by 119
  15. 2017JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correctioncited by 96
  16. 2020The Turking Test: Can Language Models Understand Instructions?cited by 82
  17. 2020Learning from Task Descriptionscited by 78
  18. 2021FLEX: Unifying Evaluation for Few-Shot NLPcited by 67
  19. 2016A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Taskscited by 62
  20. 2021CaSiNo: A Corpus of Campsite Negotiation Dialogues for Automatic Negotiation Systemscited by 60
  21. 2022ZeroPrompt: Scaling Prompt-Based Pretraining to 1,000 Tasks Improves Zero-Shot Generalizationcited by 54
  22. 2021Learning to Generate Task-Specific Adapters from Task Descriptioncited by 28
  23. 2021Disfl-QA: A Benchmark Dataset for Understanding Disfluencies in Question Answeringcited by 20
  24. 2020Author's Sentiment Predictioncited by 18
  25. 2020Understanding Points of Correspondence between Sentences for Abstractive Summarizationcited by 16
  26. 2022One-Shot Learning from a Demonstration with Hierarchical Latent Languagecited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 17; 2023: 273; 2024: 310; 2025: 177; 2026: 81.

858 papers.

Sort:

By area:

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  5. 2023A Survey of Large Language Modelscited by 2,537
  6. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  7. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  8. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  9. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  10. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  11. 2023LIMA: Less Is More for Alignmentcited by 1,080
  12. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
38 more
  1. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  2. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  3. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  4. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  5. 2022A Survey on In-context Learningcited by 717
  6. 2023A Comprehensive Overview of Large Language Modelscited by 639
  7. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  8. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  9. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  10. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  11. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  12. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  13. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  14. 2023Augmented Language Models: a Surveycited by 398
  15. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 396
  16. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  17. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  18. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  19. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  20. 2023GPTScore: Evaluate as You Desirecited by 357
  21. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  22. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  23. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  24. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  25. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 301
  26. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  27. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 296
  28. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  29. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  30. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  31. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  32. 2023Can Large Language Models Transform Computational Social Science?cited by 245
  33. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 243
  34. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 240
  35. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  36. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  37. 2024Generative Representational Instruction Tuningcited by 227
  38. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 203

50 most cited of 858 All 858 (JSON)