# StablePrompt: Automatic Prompt Tuning using Reinforcement Learning for Large Language Models

arXiv: https://arxiv.org/abs/2410.07652 · HTML: https://mlsanity.com/paper/2410.07652/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 34 references that are in the map.
- First posted on arXiv 10 Oct 2024.

## Builds on (most cited first, 22 of 22)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [Cross-Task Generalization via Natural Language Crowdsourcing Instructions](https://mlsanity.com/paper/2104.08773/) (2021)
- [Automatic Prompt Optimization with "Gradient Descent" and Beam Search](https://mlsanity.com/paper/2305.03495/) (2023)
- [RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning](https://mlsanity.com/paper/2205.12548/) (2022)
- [PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization](https://mlsanity.com/paper/2310.16427/) (2023)
- [Instruction Induction: From Few Examples to Natural Language Task Descriptions](https://mlsanity.com/paper/2205.10782/) (2022)
- [PromptBoosting: Black-Box Text Classification with Ten Forward Passes](https://mlsanity.com/paper/2212.09257/) (2022)
- [Reinforcement Learning-Based Black-Box Model Inversion Attacks](https://mlsanity.com/paper/2304.04625/) (2023)

## Built on by (most cited first, 14 of 14)

- [Prompt-MII: Meta-Learning Instruction Induction for LLMs](https://mlsanity.com/paper/2510.16932/) (2025)
- [SIPDO: Closed-Loop Prompt Optimization via Synthetic Data Feedback](https://mlsanity.com/paper/2505.19514/) (2025)
- [No Loss, No Gain: Gated Refinement and Adaptive Compression for Prompt Optimization](https://mlsanity.com/paper/2509.23387/) (2025)
- [How to Auto-optimize Prompts for Domain Tasks? Adaptive Prompting and Reasoning through Evolutionary Domain Knowledge Adaptation](https://mlsanity.com/paper/2510.21148/) (2025)
- [Adapting Generalist Robot Policies with Semantic Reinforcement Learning](https://mlsanity.com/paper/2606.31958/) (2026)
- [Bounds of Chain-of-Thought Robustness: Reasoning Steps, Embed Norms, and Beyond](https://mlsanity.com/paper/2509.21284/) (2025)
- [Scaffolded Language Models with Language Supervision for Mixed-Autonomy: A Survey](https://mlsanity.com/paper/2410.16392/) (2024)
- [Beyond Elicitation: Provision-based Prompt Optimization for Knowledge-Intensive Tasks](https://mlsanity.com/paper/2511.10465/) (2025)
- [Scaling Test-Time Robustness of Vision-Language Models via Self-Critical Inference Framework](https://mlsanity.com/paper/2603.07659/) (2026)
- [p1: Better Prompt Optimization with Fewer Prompts](https://mlsanity.com/paper/2604.08801/) (2026)
- [ContextBench: Modifying Contexts for Targeted Latent Activation](https://mlsanity.com/paper/2506.15735/) (2025)
- [ReflectivePrompt: Reflective evolution in autoprompting algorithms](https://mlsanity.com/paper/2508.18870/) (2025)
- [CAPO: Constraint-Aware Prompt Optimization for LLM Agents](https://mlsanity.com/paper/2608.16068/) (2026)
- [Policy-as-Skill: Governed LLM Decision Support with Evidence, Deterministic Control, and Audit](https://mlsanity.com/paper/2609.27087/) (2026)

## Abstract (arXiv)

Finding appropriate prompts for the specific task has become an important issue as the usage of Large Language Models (LLM) has expanded. Reinforcement Learning (RL) is widely used for prompt tuning, but its inherent instability and environmental dependency make it difficult to use in practice. In this paper, we propose StablePrompt, which strikes a balance between training stability and search space, mitigating the instability of RL and producing high-performance prompts. We formulate prompt tuning as an online RL problem between the agent and target LLM and introduce Adaptive Proximal Policy Optimization (APPO). APPO introduces an LLM anchor model to adaptively adjust the rate of policy updates. This allows for flexible prompt search while preserving the linguistic ability of the pre-trained LLM. StablePrompt outperforms previous methods on various tasks including text classification, question answering, and text generation. Our code can be found in github.