mlsanityData to 7 Oct 2026

SPT: Semi-Parametric Prompt Tuning for Multitask Prompted Learning

arXiv 2212.10929 ↗First posted 21 Dec 2022cs.CL cs.AI cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Pre-trained large language models can efficiently interpolate human-written prompts in a natural way. Multitask prompted learning can help generalization through a diverse set of tasks at once, thus enhancing the potential for more effective downstream fine-tuning. To perform efficient multitask-inference in the same batch, parameter-efficient fine-tuning methods such as prompt tuning have been proposed. However, the existing prompt tuning methods may lack generalization. We propose SPT, a semi-parametric prompt tuning method for multitask prompted learning. The novel component of SPT is a memory bank from where memory prompts are retrieved based on discrete prompts. Extensive experiments, such as (i) fine-tuning a full language model with SPT on 31 different tasks from 8 different domains and evaluating zero-shot generalization on 9 heldout datasets under 5 NLP task categories and (ii) pretraining SPT on the GLUE datasets and evaluating fine-tuning on the SuperGLUE datasets, demonstrate effectiveness of SPT.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 43 of its 57 references that are in the map. First posted on arXiv 21 Dec 2022.

Builds on

43 of 57 references.

References · 43

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
31 more
  1. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  2. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  3. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  4. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  5. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  6. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  7. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  8. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  9. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  10. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  11. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  12. 2017Learning multiple visual domains with residual adapterscited by 553
  13. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  14. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  15. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  16. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  17. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  18. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  19. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  20. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 239
  21. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  22. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  23. 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 194
  24. 2022What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cited by 149
  25. 2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
  26. 2021LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5cited by 103
  27. 2021What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformerscited by 75
  28. 2021Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with 1/n Parameterscited by 66
  29. 2022When to Use Multi-Task Learning vs Intermediate Fine-Tuning for Pre-Trained Encoder Transfer Learningcited by 17
  30. 2021Nearest Neighbour Few-Shot Learning for Cross-lingual Classificationcited by 8
  31. 2021Parameterized Hypercomplex Graph Neural Networks for Graph Classificationcited by 7

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 1.

5 papers.

  1. 2023Exploring the Benefits of Training Expert Language Models over Instruction Tuningcited by 80
  2. 2023Parameter-Efficient Fine-Tuning Design Spacescited by 67
  3. 2023Prompt Pre-Training with Twenty-Thousand Classes for Open-Vocabulary Visual Recognitioncited by 35
  4. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Taskscited by 27
  5. 2023Automated Few-shot Classification with Instruction-Finetuned Language Modelscited by 9