mlsanityData to 13 Oct 2026

Learning from Task Descriptions

arXiv 2011.08115 ↗First posted 16 Nov 2020cs.CLcited by 78

Not on the map. Why?

Abstract (arXiv)

Typically, machine learning systems solve new tasks by training on thousands of examples. In contrast, humans can solve new tasks by reading some instructions, with perhaps an example or two. To take a step toward closing this gap, we introduce a framework for developing NLP systems that solve new tasks after reading their descriptions, synthesizing prior work in this area. We instantiate this framework with a new English language dataset, ZEST, structured for task-oriented evaluation on unseen tasks. Formulating task descriptions as questions, we ensure each is general enough to apply to many possible inputs, thus comprehensively evaluating a model's ability to solve each task. Moreover, the dataset's structure tests specific types of systematic generalization. We find that the state-of-the-art T5 model achieves a score of 12% on ZEST, leaving a significant challenge for NLP researchers.

Key facts. Cited by 78 papers in the map (data through 13 Oct 2026). First posted on arXiv 16 Nov 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 20; 2022: 20; 2023: 20; 2024: 11; 2025: 7.

78 papers.

Sort:

By area:

  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,693
  2. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 865
  3. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 697
  4. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 407
  5. 2022Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)cited by 397
  6. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 363
  7. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 326
  8. 2021Generating Datasets with Pretrained Language Modelscited by 193
  9. 2021Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collectionscited by 128
  10. 2022GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Modelscited by 127
  11. 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124
  12. 2023FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Modelscited by 116
38 more
  1. 2021A Recipe For Arbitrary Text Style Transfer with Large Language Modelscited by 107
  2. 2021KaggleDBQA: Realistic Evaluation of Text-to-SQL Parserscited by 106
  3. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 98
  4. 2021Competency Problems: On Finding and Removing Artifacts in Language Datacited by 91
  5. 2021When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Datacited by 75
  6. 2023On the Calibration of Large Language Models and Alignmentcited by 72
  7. 2023Style Over Substance: Evaluation Biases for Large Language Modelscited by 69
  8. 2024FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructionscited by 68
  9. 2024LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Modelscited by 68
  10. 2021FLEX: Unifying Evaluation for Few-Shot NLPcited by 67
  11. 2024CFBench: A Comprehensive Constraints-Following Benchmark for LLMscited by 65
  12. 2024Adapting Large Language Models for Document-Level Machine Translationcited by 59
  13. 2022Improving In-Context Few-Shot Learning via Self-Supervised Trainingcited by 57
  14. 2021Towards General Purpose Vision Systemscited by 46
  15. 2021Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions?cited by 46
  16. 2023VIP5: Towards Multimodal Foundation Models for Recommendationcited by 45
  17. 2022InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuningcited by 39
  18. 2021True Few-Shot Learning with Prompts -- A Real-World Perspectivecited by 38
  19. 2021Exploiting Adapters for Cross-lingual Low-resource Speech Recognitioncited by 34
  20. 2024SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuningcited by 32
  21. 2025A Closer Look at System Prompt Robustnesscited by 32
  22. 2023Large Language Model Instruction Following: A Survey of Progresses and Challengescited by 31
  23. 2023Failures Pave the Way: Enhancing Large Language Models through Tuning-free Rule Accumulationcited by 30
  24. 2021Learning to Generate Task-Specific Adapters from Task Descriptioncited by 28
  25. 2022HINT: Hypernetwork Instruction Tuning for Efficient Zero- & Few-Shot Generalisationcited by 28
  26. 2023InstructProtein: Aligning Human and Protein Language via Knowledge Instructioncited by 27
  27. 2023InstructABSA: Instruction Learning for Aspect Based Sentiment Analysiscited by 22
  28. 2021Factoring Statutory Reasoning as Language Understanding Challengescited by 20
  29. 2024Beyond Probabilities: Unveiling the Misalignment in Evaluating Large Language Modelscited by 19
  30. 2022When to Use Multi-Task Learning vs Intermediate Fine-Tuning for Pre-Trained Encoder Transfer Learningcited by 17
  31. 2022HELP ME THINK: A Simple Prompting Strategy for Non-experts to Create Customized Content with Modelscited by 17
  32. 2022How Many Data Samples is an Additional Instruction Worth?cited by 16
  33. 2022In-BoXBART: Get Instructions into Biomedical Multi-Task Learningcited by 16
  34. 2023CrossCodeBench: Benchmarking Cross-Task Generalization of Source Code Modelscited by 16
  35. 2022Learning Instructions with Unlabeled Data for Zero-Shot Cross-Task Generalizationcited by 15
  36. 2023InstructExcel: A Benchmark for Natural Language Instruction in Excelcited by 15
  37. 2023Prompting with Pseudo-Code Instructionscited by 14
  38. 2024How Reliable Are Automatic Evaluation Methods for Instruction-Tuned LLMs?cited by 13

50 most cited of 78 All 78 (JSON)