mlsanityData to 7 Oct 2026

Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery

arXiv 2302.03668 ↗First posted 7 Feb 2023cs.LG cs.CLcited by 329

Not on the map. Why?

Cited by 97 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The strength of modern generative models lies in their ability to be controlled through text-based prompts. Typical "hard" prompts are made from interpretable words and tokens, and must be hand-crafted by humans. There are also "soft" prompts, which consist of continuous feature vectors. These can be discovered using powerful optimization methods, but they cannot be easily interpreted, re-used across models, or plugged into a text-based interface. We describe an approach to robustly optimize hard text prompts through efficient gradient-based optimization. Our approach automatically generates hard text-based prompts for both text-to-image and text-to-text applications. In the text-to-image setting, the method creates hard prompts for diffusion models, allowing API users to easily generate, discover, and mix and match image concepts without prior knowledge on how to prompt the model. In the text-to-text setting, we show that hard prompts can be automatically discovered that are effective in tuning LMs for classification.

Key facts. Cited by 329 papers in the map (data through 7 Oct 2026). Builds on 30 of its 39 references that are in the map. First posted on arXiv 7 Feb 2023.

Builds on

30 of 39 references.

References · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  12. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
18 more
  1. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  2. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  3. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  4. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  5. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  6. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  7. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  8. 2022Red-Teaming the Stable Diffusion Safety Filtercited by 257
  9. 2020AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Promptscited by 246
  10. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 233
  11. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  12. 2021OpenPrompt: An Open-source Framework for Prompt-learningcited by 148
  13. 2022GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Modelscited by 127
  14. 2017Training Quantized Nets: A Deeper Understandingcited by 74
  15. 2021Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Promptscited by 64
  16. 2022Gradient-Based Constrained Sampling from Language Modelscited by 64
  17. 2022Toward Human Readable Prompt Tuning: Kubrick's The Shining is a good movie, and a good prompt too?cited by 49
  18. 2022Discovering the Hidden Vocabulary of DALLE-2cited by 42

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 62; 2024: 118; 2025: 100; 2026: 49.

329 papers.

Sort:

By area:

  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  2. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  3. 2023Large Language Models as Optimizerscited by 830
  4. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  5. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  6. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  7. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  8. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  9. 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 270
  10. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 240
  11. 2023Ring-A-Bell! How Reliable are Concept Removal Methods for Diffusion Models?cited by 239
  12. 2023PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimizationcited by 226
38 more
  1. 2023Understanding and Mitigating Copying in Diffusion Modelscited by 205
  2. 2023To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy To Generate Unsafe Images ... For Nowcited by 204
  3. 2023Glaze: Protecting Artists from Style Mimicry by Text-to-Image Modelscited by 200
  4. 2023StyleDrop: Text-to-Image Generation in Any Stylecited by 197
  5. 2024Defensive Unlearning with Adversarial Training for Robust Concept Erasure in Diffusion Modelscited by 164
  6. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 162
  7. 2023Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic Promptscited by 161
  8. 2024Jailbreaking Attack against Multimodal Large Language Modelcited by 128
  9. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  10. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratchcited by 111
  11. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Modelscited by 102
  12. 2023Towards image compression with perfect realism at ultra-low bitratescited by 99
  13. 2024Improved Techniques for Optimization-Based Jailbreaking on Large Language Modelscited by 98
  14. 2024Detecting, Explaining, and Mitigating Memorization in Diffusion Modelscited by 94
  15. 2024Attacking Large Language Models with Projected Gradient Descentcited by 91
  16. 2023Expressive Text-to-Image Generation with Rich Textcited by 89
  17. 2024Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Spacecited by 88
  18. 2024Coercing LLMs to do and reveal (almost) anythingcited by 88
  19. 2024Fast Adversarial Attacks on Language Models In One GPU Minutecited by 85
  20. 2023A Pilot Study of Query-Free Adversarial Attack against Stable Diffusioncited by 75
  21. 2023ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Modelscited by 75
  22. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  23. 2024Defending Large Language Models against Jailbreak Attacks via Semantic Smoothingcited by 68
  24. 2024Jailbreaking Prompt Attack: A Controllable Adversarial Attack against Diffusion Modelscited by 68
  25. 2024Latent Guard: a Safety Framework for Text-to-image Generationcited by 68
  26. 2023Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Modelscited by 67
  27. 2024Imprompter: Tricking LLM Agents into Improper Tool Usecited by 66
  28. 2023AntifakePrompt: Prompt-Tuned Vision-Language Models are Fake Image Detectorscited by 64
  29. 2023Text + Sketch: Image Compression at Ultra Low Ratescited by 63
  30. 2024PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinementcited by 61
  31. 2024Yo'LLaVA: Your Personalized Language and Vision Assistantcited by 61
  32. 2023ImageBrush: Learning Visual In-Context Instructions for Exemplar-Based Image Manipulationcited by 59
  33. 2023DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineercited by 56
  34. 2024R.A.C.E.: Robust Adversarial Concept Erasure for Secure Text-to-Image Diffusion Modelcited by 56
  35. 2024Don't Say No: Jailbreaking LLM by Suppressing Refusalcited by 53
  36. 2024MC-LLaVA: Multi-Concept Personalized Vision-Language Modelcited by 53
  37. 2024DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processingcited by 49
  38. 2024Efficient Prompting Methods for Large Language Models: A Surveycited by 46

50 most cited of 329 All 329 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Prompt Diffusion Robustifies Any-Modality Prompt Learning
  2. 2024Batch-Instructed Gradient for Prompt Evolution:Systematic Prompt Optimization for Enhanced Text-to-Image Synthesis
  3. 2026Evolutionary Token-Level Prompt Optimization for Diffusion Models
  4. 2022Optimizing Prompts for Text-to-Image Generation
  5. 2025ConceptMix++: Leveling the Playing Field in Text-to-Image Benchmarking via Iterative Prompt Optimization
  6. 2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models
  7. 2025Reward-Agnostic Prompt Optimization for Text-to-Image Diffusion Models
  8. 2026ELROND: Exploring and decomposing intrinsic capabilities of diffusion models