mlsanityData to 7 Oct 2026

GPT Understands, Too

arXiv 2103.10385 ↗First posted 18 Mar 2021cs.CL cs.LGcited by 715

Not on the map. Why?

Cited by 38 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Prompting a pretrained language model with natural language patterns has been proved effective for natural language understanding (NLU). However, our preliminary study reveals that manual discrete prompts often lead to unstable performance -- e.g., changing a single word in the prompt might result in substantial performance drop. We propose a novel method P-Tuning that employs trainable continuous prompt embeddings in concatenation with discrete prompts. Empirically, P-Tuning not only stabilizes training by minimizing the gap between various discrete prompts, but also improves performance by a sizeable margin on a wide range of NLU tasks including LAMA and SuperGLUE. P-Tuning is generally effective for both frozen and tuned language models, under both the fully-supervised and few-shot settings.

Key facts. Cited by 715 papers in the map (data through 7 Oct 2026). Builds on 31 of its 48 references that are in the map. First posted on arXiv 18 Mar 2021.

Builds on

31 of 48 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  10. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  11. 2019Language Models as Knowledge Bases?cited by 1,922
  12. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
19 more
  1. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  2. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  3. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  4. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  5. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  6. 2019How Can We Know What Language Models Know?cited by 953
  7. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  8. 2020Self-supervised Learning: Generative or Contrastivecited by 693
  9. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  10. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  11. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  12. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  13. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  14. 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 198
  15. 2019Commonsense Knowledge Mining from Pretrained Modelscited by 189
  16. 2020How Context Affects Language Models' Factual Predictionscited by 181
  17. 2020X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Modelscited by 117
  18. 2020Interventional Few-Shot Learningcited by 112
  19. 2020Language Models are Open Knowledge Graphscited by 95

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 48; 2022: 168; 2023: 242; 2024: 149; 2025: 82; 2026: 26.

715 papers.

Sort:

By area:

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  3. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  4. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  5. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  6. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  7. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  8. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  9. 2023Large Language Models as Optimizerscited by 830
  10. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  11. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  12. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
38 more
  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  2. 2023A Comprehensive Overview of Large Language Modelscited by 639
  3. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  4. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  5. 2022Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)cited by 396
  6. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  7. 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 345
  8. 2022Prompt-aligned Gradient for Prompt Tuningcited by 339
  9. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
  10. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  11. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 273
  12. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 271
  13. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  14. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 254
  15. 2021PTR: Prompt Tuning with Rules for Text Classificationcited by 238
  16. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  17. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  18. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
  19. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  20. 2023EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizerscited by 189
  21. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 184
  22. 2021Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classificationcited by 183
  23. 2023Efficient Large Language Models: A Surveycited by 183
  24. 2021KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extractioncited by 180
  25. 2021Reframing Instructional Prompts to GPTk's Languagecited by 176
  26. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 171
  27. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
  28. 2023GraphPrompt: Unifying Pre-Training and Downstream Tasks for Graph Neural Networkscited by 166
  29. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 163
  30. 2022A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunitiescited by 162
  31. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  32. 2023A Study of Generative Large Language Model for Medical Research and Healthcarecited by 160
  33. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMscited by 160
  34. 2023Active Prompting with Chain-of-Thought for Large Language Modelscited by 154
  35. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 154
  36. 2023Large Language Models for Code: Security Hardening and Adversarial Testingcited by 149
  37. 2021OpenPrompt: An Open-source Framework for Prompt-learningcited by 148
  38. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146

50 most cited of 715 All 715 (JSON)