Data to 7 Oct 2026

AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts

arXiv 2010.15980 ↗First posted 29 Oct 2020cs.CL cs.LGcited by 246

Not on the map. Why?

Abstract (arXiv)

The remarkable success of pretrained language models has motivated the study of what kinds of knowledge these models learn during pretraining. Reformulating tasks as fill-in-the-blanks problems (e.g., cloze tests) is a natural approach for gauging such knowledge, however, its usage is limited by the manual effort and guesswork required to write suitable prompts. To address this, we develop AutoPrompt, an automated method to create prompts for a diverse set of tasks, based on a gradient-guided search. Using AutoPrompt, we show that masked language models (MLMs) have an inherent capability to perform sentiment analysis and natural language inference without additional parameters or finetuning, sometimes achieving performance on par with recent state-of-the-art supervised models. We also show that our prompts elicit more accurate factual knowledge from MLMs than the manually created prompts on the LAMA benchmark, and that MLMs can be used as relation extractors more effectively than supervised relation extraction models. These results demonstrate that automatically generated prompts are a viable parameter-free alternative to existing probing methods, and as pretrained LMs become more sophisticated and capable, potentially a replacement for finetuning.

Key facts. Cited by 246 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 32; 2022: 65; 2023: 64; 2024: 44; 2025: 38; 2026: 3.

246 papers.

Sort:

By area:

2021Learning to Prompt for Vision-Language Modelscited by 2,269
2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 310
2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288

38 more

2021AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Promptscited by 265
2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 263
2022Black-Box Tuning for Language-Model-as-a-Servicecited by 254
2023Poisoning Language Models During Instruction Tuningcited by 254
2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
2021PTR: Prompt Tuning with Rules for Text Classificationcited by 238
2023Evaluating the Ripple Effects of Knowledge Editing in Language Modelscited by 238
2023Knowledge Editing for Large Language Models: A Surveycited by 189
2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
2023Efficient Large Language Models: A Surveycited by 183
2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Modelscited by 182
2021KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extractioncited by 180
2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
2021Reframing Instructional Prompts to GPTk's Languagecited by 176
2023All in One: Multi-task Prompting for Graph Neural Networkscited by 147
2022Prompting PaLM for Translation: Assessing Strategies and Performancecited by 144
2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
2023DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4cited by 128
2023Open Sesame! Universal Black Box Jailbreaking of Large Language Modelscited by 128
2022Task Residual for Tuning Vision-Language Modelscited by 122
2022Pretrained Language Models for Text Generation: A Surveycited by 121
2022CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailmentcited by 113
2022Repository-Level Prompt Generation for Large Language Models of Codecited by 106
2023Guiding Large Language Models via Directional Stimulus Promptingcited by 94
2021Language Models are Few-shot Multilingual Learnerscited by 92
2022SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Modelscited by 90
2021BERTese: Learning to Speak to BERTcited by 82
2022Efficient Methods for Natural Language Processing: A Surveycited by 78
2024Neural Exec: Learning (and Learning from) Execution Triggers for Prompt Injection Attackscited by 78
2022Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Modelscited by 67
2022HyperPrompt: Prompt-based Task-Conditioning of Transformerscited by 67
2021Few-Shot Bot: Prompt-Based Learning for Dialogue Systemscited by 64
2021Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Promptscited by 64
2021Prompt-Learning for Fine-Grained Entity Typingcited by 62
2023GeoLLM: Extracting Geospatial Knowledge from Large Language Modelscited by 62
2021PADA: Example-based Prompt Learning for on-the-fly Adaptation to Unseen Domainscited by 61
2023The Wisdom of Hindsight Makes Language Models Better Instruction Followerscited by 59

50 most cited of 246 All 246 (JSON)