mlsanityData to 14 Oct 2026

Automatic Label Sequence Generation for Prompting Sequence-to-sequence Models

arXiv 2209.09401 ↗First posted 20 Sep 2022cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Prompting, which casts downstream applications as language modeling tasks, has shown to be sample efficient compared to standard fine-tuning with pre-trained models. However, one pitfall of prompting is the need of manually-designed patterns, whose outcome can be unintuitive and requires large validation sets to tune. To tackle the challenge, we propose AutoSeq, a fully automatic prompting method: (1) We adopt natural language prompts on sequence-to-sequence models, enabling free-form generation and larger label search space; (2) We propose label sequences -- phrases with indefinite lengths to verbalize the labels -- which eliminate the need of manual templates and are more expressive than single label words; (3) We use beam search to automatically generate a large amount of label sequence candidates and propose contrastive re-ranking to get the best combinations. AutoSeq significantly outperforms other no-manual-design methods, such as soft prompt tuning, adapter tuning, and automatic search on single label words; the generated label sequences are even better than curated manual ones on a variety of tasks. Our method reveals the potential of sequence-to-sequence models in few-shot learning and sheds light on a path to generic and automatic prompting. The source code of this paper can be obtained from https://github.com/thunlp/Seq2Seq-Prompt.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 53 references that are in the map. First posted on arXiv 20 Sep 2022.

Builds on

31 of 53 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  10. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  11. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  12. 2020GLU Variants Improve Transformercited by 1,804
19 more
  1. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  2. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  3. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  4. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  5. 2019How Can We Know What Language Models Know?cited by 959
  6. 2018Neural Network Acceptability Judgmentscited by 958
  7. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
  8. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  9. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 669
  10. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 372
  11. 2021True Few-Shot Learning with Language Modelscited by 344
  12. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  13. 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 200
  14. 2019Commonsense Knowledge Mining from Pretrained Modelscited by 189
  15. 2021Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Modelscited by 139
  16. 2021Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learnerscited by 115
  17. 2020Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classificationcited by 98
  18. 2021BERTese: Learning to Speak to BERTcited by 82
  19. 2022PERFECT: Prompt-free and Efficient Few-shot Learning with Language Modelscited by 40

Built on by

Not cited yet.