mlsanityData to 7 Oct 2026

Prompt Tuning for Discriminative Pre-trained Language Models

arXiv 2205.11166 ↗First posted 23 May 2022cs.CL cs.AIcited by 17

Not on the map. Why?

Abstract (arXiv)

Recent works have shown promising results of prompt tuning in stimulating pre-trained language models (PLMs) for natural language processing (NLP) tasks. However, to the best of our knowledge, existing works focus on prompt-tuning generative PLMs that are pre-trained to generate target tokens, such as BERT. It is still unknown whether and how discriminative PLMs, e.g., ELECTRA, can be effectively prompt-tuned. In this work, we present DPT, the first prompt tuning framework for discriminative PLMs, which reformulates NLP tasks into a discriminative language modeling problem. Comprehensive experiments on text classification and question answering show that, compared with vanilla fine-tuning, DPT achieves significantly higher performance, and also prevents the unstable problem in tuning large PLMs in both full-set and low-resource settings. The source code and experiment details of this paper can be obtained from https://github.com/thunlp/DPT.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 21 of its 28 references that are in the map. First posted on arXiv 23 May 2022.

Builds on

21 of 28 references.

References · 21

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  7. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  8. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  9. 2019Language Models as Knowledge Bases?cited by 1,922
  10. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  11. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  12. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
9 more
  1. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  2. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  3. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  4. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  5. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 214
  6. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Modelscited by 182
  7. 2019Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Modelcited by 102
  8. 2019Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoningcited by 100
  9. 2020Coreferential Reasoning Learning for Language Representationcited by 80

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 5; 2024: 2; 2025: 2; 2026: 1.

17 papers.

Sort:

By area:

  1. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  2. 2023Language Models as Black-Box Optimizers for Vision-Language Modelscited by 36
  3. 2023AfriSpeech-200: Pan-African Accented Speech Dataset for Clinical and General Domain ASRcited by 34
  4. 2022Efficient Vision-Language Pretraining with Visual Concepts and Hierarchical Alignmentcited by 19
  5. 2023Building Efficient Universal Classifiers with Natural Language Inferencecited by 19
  6. 2022Cold-Start Data Selection for Few-shot Language Model Fine-tuning: A Prompt-Based Uncertainty Propagation Approachcited by 16
  7. 2023LM-CPPF: Paraphrasing-Guided Data Augmentation for Contrastive Prompt-Based Few-Shot Fine-Tuningcited by 15
  8. 2022A Universal Discriminator for Zero-Shot Generalizationcited by 11
  9. 2022Continuous Prompt Tuning Based Textual Entailment Model for E-commerce Entity Typingcited by 5
  10. 2024FineDiffusion: Scaling up Diffusion Models for Fine-grained Image Generation with 10,000 Classescited by 5
  11. 2022ELECTRA is a Zero-Shot Learner, Toocited by 4
  12. 2025Generalizable Prompt Learning of CLIP: A Brief Overviewcited by 3
5 more
  1. 2022Prompting ELECTRA: Few-Shot Learning with Discriminative Pre-Trained Modelscited by 2
  2. 2022Discriminative Language Model as Semantic Consistency Scorer for Prompt-based Few-Shot Text Classificationcited by 0
  3. 2024Developing an End-to-End Framework for Predicting the Social Communication Severity Scores of Children with Autism Spectrum Disordercited by 0
  4. 2025Exploring the Use of Contrastive Language-Image Pre-Training for Human Posture Classification: Insights from Yoga Pose Analysiscited by 0
  5. 2026Efficient Prompt Learning for Traffic Forecastingcited by 0