mlsanityData to 7 Oct 2026

Improving Cross-Task Generalization with Step-by-Step Instructions

arXiv 2305.04429 ↗First posted 8 May 2023cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

Instruction tuning has been shown to be able to improve cross-task generalization of language models. However, it is still challenging for language models to complete the target tasks following the instructions, as the instructions are general and lack intermediate steps. To address this problem, we propose to incorporate the step-by-step instructions to help language models to decompose the tasks, which can provide the detailed and specific procedures for completing the target tasks. The step-by-step instructions are obtained automatically by prompting ChatGPT, which are further combined with the original instructions to tune language models. The extensive experiments on SUP-NATINST show that the high-quality step-by-step instructions can improve cross-task generalization across different model sizes. Moreover, the further analysis indicates the importance of the order of steps of the step-by-step instruction for the improvement. To facilitate future research, we release the step-by-step instructions and their human quality evaluation results.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 19 of its 31 references that are in the map. First posted on arXiv 8 May 2023.

Builds on

19 of 31 references.

References · 19

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  10. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  11. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  12. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
7 more
  1. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  2. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  3. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  4. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  5. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  6. 2021Reframing Instructional Prompts to GPTk's Languagecited by 176
  7. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 4.

7 papers.

  1. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questionscited by 125
  2. 2024Effectiveness Assessment of Recent Large Vision-Language Modelscited by 29
  3. 2024Towards Understanding Multi-Task Learning (Generalization) of LLMs via Detecting and Exploring Task-Specific Neuronscited by 23
  4. 2023LayoutNUWA: Revealing the Hidden Layout Expertise of Large Language Modelscited by 17
  5. 2023OpenBA: An Open-sourced 15B Bilingual Asymmetric seq2seq Model Pre-trained from Scratchcited by 7
  6. 2024In-Context Translation: Towards Unifying Image Recognition, Processing, and Generationcited by 0
  7. 2024Patch is Enough: Naturalistic Adversarial Patch against Vision-Language Pre-training Modelscited by 0