mlsanityData to 7 Oct 2026

Cross-Task Generalization via Natural Language Crowdsourcing Instructions

arXiv 2104.08773 ↗First posted 18 Apr 2021cs.CL cs.AI cs.CVcited by 696

Not on the map. Why?

Cited by 55 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Humans (e.g., crowdworkers) have a remarkable ability in solving different tasks, by simply reading textual instructions that define them and looking at a few examples. Despite the success of the conventional supervised learning on individual datasets, such models often struggle with generalization across tasks (e.g., a question-answering system cannot solve classification tasks). A long-standing challenge in AI is to build a model that learns a new task by understanding the human-readable instructions that define it. To study this, we introduce NATURAL INSTRUCTIONS, a dataset of 61 distinct tasks, their human-authored instructions, and 193k task instances (input-output pairs). The instructions are obtained from crowdsourcing instructions used to create existing NLP datasets and mapped to a unified schema. Using this meta-dataset, we measure cross-task generalization by training models on seen tasks and measuring generalization to the remaining unseen ones. We adopt generative pre-trained language models to encode task-specific instructions along with input and generate task output. Our results indicate that models benefit from instructions when evaluated in terms of generalization to unseen tasks (19% better for models utilizing instructions). These models, however, are far behind an estimated performance upperbound indicating significant room for more progress in this direction.

Key facts. Cited by 696 papers in the map (data through 7 Oct 2026). Builds on 32 of its 55 references that are in the map. First posted on arXiv 18 Apr 2021.

Builds on

32 of 55 references.

References · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2020Longformer: The Long-Document Transformercited by 3,362
  5. 2018Deep contextualized word representationscited by 2,285
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  7. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  8. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  9. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  10. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  11. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  12. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
20 more
  1. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 281
  2. 2019QASC: A Dataset for Question Answering via Sentence Compositioncited by 248
  3. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 235
  4. 2021How Many Data Points is a Prompt Worth?cited by 197
  5. 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 194
  6. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  7. 2020Language-Conditioned Imitation Learning for Robot Manipulation Taskscited by 136
  8. 2021Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collectionscited by 128
  9. 2019"Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Commonsense Understandingcited by 121
  10. 2019Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoningcited by 100
  11. 2018NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating Systemcited by 97
  12. 2021Improving and Simplifying Pattern Exploiting Trainingcited by 97
  13. 2020Few-Shot Text Generation with Pattern-Exploiting Trainingcited by 87
  14. 2020Hard negative examples are hard, but usefulcited by 82
  15. 2020The Turking Test: Can Language Models Understand Instructions?cited by 82
  16. 2019Reasoning Over Paragraph Effects in Situationscited by 79
  17. 2020Learning from Task Descriptionscited by 78
  18. 2021When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Datacited by 75
  19. 2021Towards General Purpose Vision Systemscited by 46
  20. 2020Towards Question Format Independent Numerical Reasoning: A Set of Prerequisite Taskscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 88; 2023: 219; 2024: 231; 2025: 103; 2026: 45.

696 papers.

Sort:

By area:

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  6. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  7. 2023A Survey of Large Language Modelscited by 2,537
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  9. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  10. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  11. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  12. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
38 more
  1. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  2. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  3. 2022Editing Models with Task Arithmeticcited by 1,076
  4. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  5. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  6. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  7. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  8. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  9. 2022A Survey on In-context Learningcited by 717
  10. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  11. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  12. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  13. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  14. 2024Large Language Models: A Surveycited by 506
  15. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  16. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  17. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  18. 2021MetaICL: Learning to Learn In Contextcited by 450
  19. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  20. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  21. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Modelcited by 376
  22. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  23. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  24. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  25. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  26. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
  27. 2023Tool Learning with Foundation Modelscited by 302
  28. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 299
  29. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  30. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  31. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  32. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  33. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  34. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 273
  35. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  36. 2021AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Promptscited by 265
  37. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 258
  38. 2022Can language models learn from explanations in context?cited by 248

50 most cited of 696 All 696 (JSON)