Toward Zero-Shot Instruction Following
Not on the map. Why?
Abstract (arXiv)
This work proposes a challenging yet more realistic setting for zero-shot cross-task generalization: zero-shot instruction following, presuming the existence of a paragraph-style task definition while no demonstrations exist. To better learn the task supervision from the definition, we propose two strategies: first, to automatically find out the critical sentences in the definition; second, a ranking objective to force the model to generate the gold outputs with higher probabilities when those critical parts are highlighted in the definition. The joint efforts of the two strategies yield state-of-the-art performance on the Super-NaturalInstructions. Our code is available on GitHub.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 31 of its 48 references that are in the map. First posted on arXiv 4 Aug 2023.
Builds on
31 of 48 references.
References · 31
- 2017Attention Is All You Needcited by 60,361
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
- 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
- 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,292
- 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,148
- 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
- 2015Pointer Networkscited by 895
- 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
19 more
- 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
- 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
- 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 624
- 2021MetaICL: Learning to Learn In Contextcited by 454
- 2016SeqGAN: Sequence Generative Adversarial Nets with Policy Gradientcited by 376
- 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 341
- 2022Can language models learn from explanations in context?cited by 249
- 2021KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extractioncited by 180
- 2021Reframing Instructional Prompts to GPTk's Languagecited by 176
- 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 165
- 2020Few-Shot Text Generation with Pattern-Exploiting Trainingcited by 87
- 2021Towards Interpreting and Mitigating Shortcut Learning Behavior of NLU Modelscited by 78
- 2019Ranked List Loss for Deep Metric Learningcited by 77
- 2020Recognizing Emotion Cause in Conversationscited by 66
- 2022Unsupervised Cross-Task Generalization via Retrieval Augmentationcited by 42
- 2021Contributions of Transformer Attention Heads in Multi- and Cross-lingual Taskscited by 16
- 2021GradTS: A Gradient-Based Automatic Auxiliary Task Selection Method Based on Transformer Networkscited by 7
- 2022MORE: A Metric Learning Based Framework for Open-domain Relation Extractioncited by 5
- 2019A Study of the Effect of Resolving Negation and Sentiment Analysis in Recognizing Text Entailment for Arabiccited by 2
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.