mlsanityData to 14 Oct 2026

RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning

arXiv 2205.12548 ↗First posted 25 May 2022cs.CL cs.LGcited by 366

Not on the map. Why?

Cited by 35 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Prompting has shown impressive success in enabling large pretrained language models (LMs) to perform diverse NLP tasks, especially when only few downstream data are available. Automatically finding the optimal prompt for each task, however, is challenging. Most existing work resorts to tuning soft prompt (e.g., embeddings) which falls short of interpretability, reusability across LMs, and applicability when gradients are not accessible. Discrete prompt, on the other hand, is difficult to optimize, and is often created by "enumeration (e.g., paraphrasing)-then-selection" heuristics that do not explore the prompt space systematically. This paper proposes RLPrompt, an efficient discrete prompt optimization approach with reinforcement learning (RL). RLPrompt formulates a parameter-efficient policy network that generates the desired discrete prompt after training with reward. To overcome the complexity and stochasticity of reward signals by the large LM environment, we incorporate effective reward stabilization that substantially enhances the training efficiency. RLPrompt is flexibly applicable to different types of LMs, such as masked (e.g., BERT) and left-to-right models (e.g., GPTs), for both classification and generation tasks. Experiments on few-shot classification and unsupervised text style transfer show superior performance over a wide range of existing finetuning or prompting methods. Interestingly, the resulting optimized prompts are often ungrammatical gibberish text; and surprisingly, those gibberish prompts are transferrable between different LMs to retain significant performance, indicating LM prompting may not follow human language patterns.

Key facts. Cited by 366 papers in the map (data through 14 Oct 2026). Builds on 85 of its 139 references that are in the map. First posted on arXiv 25 May 2022.

Builds on

85 of 139 references.

References · 85

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  8. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
38 more
  1. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  2. 2018Deep contextualized word representationscited by 2,290
  3. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  4. 2019Language Models as Knowledge Bases?cited by 1,936
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  6. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  7. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  8. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  9. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  10. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  11. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  12. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  13. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  14. 2022Red Teaming Language Models with Language Modelscited by 1,026
  15. 2019How Can We Know What Language Models Know?cited by 959
  16. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
  17. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  18. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 762
  19. 2017Deep Reinforcement Learning that Matterscited by 754
  20. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  21. 2021GPT Understands, Toocited by 716
  22. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  23. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 689
  24. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 669
  25. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
  26. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 441
  27. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 377
  28. 2016SeqGAN: Sequence Generative Adversarial Nets with Policy Gradientcited by 376
  29. 2021True Few-Shot Learning with Language Modelscited by 344
  30. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 326
  31. 2017Toward Controlled Generation of Textcited by 272
  32. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 256
  33. 2017Style Transfer from Non-Parallel Text by Cross-Alignmentcited by 248
  34. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  35. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  36. 2021WARP: Word-level Adversarial ReProgrammingcited by 221
  37. 2018Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfercited by 191
  38. 2020Reformulating Unsupervised Style Transfer as Paraphrase Generationcited by 183

50 most cited of 85 All 85 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 12; 2023: 87; 2024: 117; 2025: 82; 2026: 68.

366 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,576
  2. 2023Large Language Models as Optimizerscited by 864
  3. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  4. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 653
  5. 2023Fine-Tuning Language Models with Just Forward Passescited by 345
  6. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 336
  7. 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 291
  8. 2023PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimizationcited by 233
  9. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 216
  10. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 214
  11. 2023EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizerscited by 193
  12. 2023Implicit Chain of Thought Reasoning via Knowledge Distillationcited by 176
38 more
  1. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 175
  2. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 161
  3. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 156
  4. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 135
  5. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 134
  6. 2024Personalized Language Modeling from Personalized Human Feedbackcited by 132
  7. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 126
  8. 2024On Prompt-Driven Safeguarding for Large Language Modelscited by 116
  9. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratchcited by 114
  10. 2023RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputscited by 105
  11. 2023Guiding Large Language Models via Directional Stimulus Promptingcited by 95
  12. 2023Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineeringcited by 94
  13. 2023Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Modelscited by 91
  14. 2023Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMscited by 91
  15. 2024Language Agents as Optimizable Graphscited by 89
  16. 2024Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?cited by 81
  17. 2023VPGTrans: Transfer Visual Prompt Generator across LLMscited by 80
  18. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
  19. 2024AvaTaR: Optimizing LLM Agents for Tool Usage via Contrastive Reasoningcited by 72
  20. 2023BlackVIP: Black-Box Visual Prompting for Robust Transfer Learningcited by 70
  21. 2024Aligning LLM Agents by Learning Latent Preference from User Editscited by 69
  22. 2023InstructZero: Efficient Instruction Optimization for Black-Box Large Language Modelscited by 68
  23. 2023Parameter-Efficient Fine-Tuning Design Spacescited by 67
  24. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wiselycited by 66
  25. 2025Effectively Controlling Reasoning Models through Thinking Interventioncited by 65
  26. 2023AntifakePrompt: Prompt-Tuned Vision-Language Models are Fake Image Detectorscited by 64
  27. 2025A Systematic Survey of Automatic Prompt Optimization Techniquescited by 63
  28. 2023The Wisdom of Hindsight Makes Language Models Better Instruction Followerscited by 62
  29. 2024POSIX: A Prompt Sensitivity Index For Large Language Modelscited by 59
  30. 2023DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineercited by 57
  31. 2024A Dense Reward View on Aligning Text-to-Image Diffusion with Preferencecited by 55
  32. 2023Prompt Engineering a Prompt Engineercited by 53
  33. 2023Better Zero-Shot Reasoning with Self-Adaptive Promptingcited by 52
  34. 2022TEMPERA: Test-Time Prompting via Reinforcement Learningcited by 50
  35. 2022Toward Human Readable Prompt Tuning: Kubrick's The Shining is a good movie, and a good prompt too?cited by 49
  36. 2023A Practical Survey on Zero-shot Prompt Design for In-context Learningcited by 49
  37. 2024Value Augmented Sampling for Language Model Alignment and Personalizationcited by 49
  38. 2025Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attackscited by 49

50 most cited of 366 All 366 (JSON)