mlsanityData to 14 Oct 2026

Learning a Better Initialization for Soft Prompts via Meta-Learning

arXiv 2205.12471 ↗First posted 25 May 2022cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

Prompt tuning (PT) is an effective approach to adapting pre-trained language models to downstream tasks. Without a good initialization, prompt tuning doesn't perform well under few-shot settings. So pre-trained prompt tuning (PPT) is proposed to initialize prompts by leveraging pre-training data. We propose MetaPT (Meta-learned Prompt Tuning) to further improve PPT's initialization by considering latent structure within the pre-training data. Specifically, we introduce the structure by first clustering pre-training data into different auxiliary tasks with unsupervised methods. Then we use these tasks to pre-train prompts with a meta-learning algorithm. Such a process can make prompts learn a better initialization by discovering commonalities among these auxiliary tasks. We evaluate our method on seven downstream tasks. Our MetaPT achieves better and more stable performance than the state-of-the-art method.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 27 of its 37 references that are in the map. First posted on arXiv 25 May 2022.

Builds on

27 of 37 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  9. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  10. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  11. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  12. 2018On First-Order Meta-Learning Algorithmscited by 1,018
15 more
  1. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  2. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 689
  3. 2019Meta-Learning with Differentiable Convex Optimizationcited by 461
  4. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 445
  5. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  6. 2018Task-Agnostic Meta-Learning for Few-shot Learningcited by 151
  7. 2018Meta-Learning for Low-Resource Neural Machine Translationcited by 105
  8. 2020Text-to-Text Pre-Training for Data-to-Text Taskscited by 104
  9. 2021BERTese: Learning to Speak to BERTcited by 82
  10. 2021PADA: Example-based Prompt Learning for on-the-fly Adaptation to Unseen Domainscited by 61
  11. 2019Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Taskscited by 48
  12. 2020Meta-learning for Few-shot Natural Language Processing: A Surveycited by 42
  13. 2019Domain Adaptive Dialog Generation via Meta Learningcited by 41
  14. 2016SentiHood: Targeted Aspect Based Sentiment Analysis Dataset for Urban Neighbourhoodscited by 32
  15. 2021A Student-Teacher Architecture for Dialog Domain Adaptation under the Meta-Learning Settingcited by 5

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 1; 2025: 5.

9 papers.

  1. 2025System Prompt Optimization with Meta-Learningcited by 18
  2. 2023Self-supervised Meta-Prompt Learning with Meta-Gradient Regularization for Few-shot Generalizationcited by 15
  3. 2023Learning to Initialize: Can Meta Learning Improve Cross-task Generalization in Prompt Tuning?cited by 11
  4. 2024PII-Scope: A Comprehensive Study on Training Data PII Extraction Attacks in LLMscited by 7
  5. 2023Black-box Prompt Tuning with Subspace Learningcited by 5
  6. 2025Unified Game Moderation: Soft-Prompting and LLM-Assisted Label Transfer for Resource-Efficient Toxicity Detectioncited by 5
  7. 2025Context Tuning for In-Context Optimizationcited by 3
  8. 2025GLAD: Generalizable Tuning for Vision-Language Modelscited by 3
  9. 2025Learning Optimal Prompt Ensemble for Multi-source Visual Prompt Transfercited by 0