# PACE: Improving Prompt with Actor-Critic Editing for Large Language Model

arXiv: https://arxiv.org/abs/2308.10088 · HTML: https://mlsanity.com/paper/2308.10088/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 41 references that are in the map.
- First posted on arXiv 19 Aug 2023.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [Automatic Prompt Optimization with "Gradient Descent" and Beam Search](https://mlsanity.com/paper/2305.03495/) (2023)
- [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://mlsanity.com/paper/2102.07350/) (2021)
- [Learning How to Ask: Querying LMs with Mixtures of Soft Prompts](https://mlsanity.com/paper/2104.06599/) (2021)
- [RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning](https://mlsanity.com/paper/2205.12548/) (2022)
- [Self-collaboration Code Generation via ChatGPT](https://mlsanity.com/paper/2304.07590/) (2023)
- [WARP: Word-level Adversarial ReProgramming](https://mlsanity.com/paper/2101.00121/) (2021)
- [Optimizing Prompts for Text-to-Image Generation](https://mlsanity.com/paper/2212.09611/) (2022)
- [Instruction Induction: From Few Examples to Natural Language Task Descriptions](https://mlsanity.com/paper/2205.10782/) (2022)
- [GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Models](https://mlsanity.com/paper/2203.07281/) (2022)
- [Self-planning Code Generation with Large Language Models](https://mlsanity.com/paper/2303.06689/) (2023)
- [EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RL](https://mlsanity.com/paper/2206.09674/) (2022)
- [Incorporating Domain Knowledge through Task Augmentation for Front-End JavaScript Code Generation](https://mlsanity.com/paper/2208.10091/) (2022)
- [Learning Answer Generation using Supervision from Automatic Question Answering Evaluators](https://mlsanity.com/paper/2305.15344/) (2023)

## Built on by (most cited first, 20 of 20)

- [Efficient Prompting Methods for Large Language Models: A Survey](https://mlsanity.com/paper/2404.01077/) (2024)
- [LLM-Empowered State Representation for Reinforcement Learning](https://mlsanity.com/paper/2407.13237/) (2024)
- [ProRefine: Inference-Time Prompt Refinement with Textual Feedback](https://mlsanity.com/paper/2506.05305/) (2025)
- [No Loss, No Gain: Gated Refinement and Adaptive Compression for Prompt Optimization](https://mlsanity.com/paper/2509.23387/) (2025)
- [SELU: Self-Learning Embodied MLLMs in Unknown Environments](https://mlsanity.com/paper/2410.03303/) (2024)
- [LogicGuard: Improving Embodied LLM agents through Temporal Logic based Critics](https://mlsanity.com/paper/2507.03293/) (2025)
- [MultiPrompter: Cooperative Prompt Optimization with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2310.16730/) (2023)
- [Exploring Data-Efficient Adaptation of Large Language Models for Code Generation](https://mlsanity.com/paper/2403.00046/) (2024)
- [Bounds of Chain-of-Thought Robustness: Reasoning Steps, Embed Norms, and Beyond](https://mlsanity.com/paper/2509.21284/) (2025)
- [Enhancing Illicit Activity Detection using XAI: A Multimodal Graph-LLM Framework](https://mlsanity.com/paper/2310.13787/) (2023)
- [Grammar-Guided Evolutionary Search for Discrete Prompt Optimisation](https://mlsanity.com/paper/2507.10326/) (2025)
- [GreenTEA: Gradient Descent with Topic-modeling and Evolutionary Auto-prompting](https://mlsanity.com/paper/2508.16603/) (2025)
- [Textual Equilibrium Propagation for Deep Compound AI Systems](https://mlsanity.com/paper/2601.21064/) (2026)
- [An Overview and Discussion on Using Large Language Models for Implementation Generation of Solutions to Open-Ended Problems](https://mlsanity.com/paper/2501.00562/) (2024)
- [Context information can be more important than reasoning for time series forecasting with a large language model](https://mlsanity.com/paper/2502.05699/) (2025)
- [Multi-Sample Prompting and Actor-Critic Prompt Optimization for Diverse Synthetic Data Generation](https://mlsanity.com/paper/2506.21138/) (2025)
- [Improving Collaborative Storytelling with a Multi-Agent Framework Based on Large Language Models](https://mlsanity.com/paper/2605.29625/) (2026)
- [A Test-time Actor-Critic Approach to News Images Generation](https://mlsanity.com/paper/2606.21304/) (2026)
- [Towards Spec Learning: Inference-Time Alignment from Preference Pairs](https://mlsanity.com/paper/2606.24004/) (2026)
- [TRACE: TRajectory Attribution for Automated Context Engineering](https://mlsanity.com/paper/2608.09153/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have showcased remarkable potential across various tasks by conditioning on prompts. However, the quality of different human-written prompts leads to substantial discrepancies in LLMs' performance, and improving prompts usually necessitates considerable human effort and expertise. To this end, this paper proposes Prompt with Actor-Critic Editing (PACE) for LLMs to enable automatic prompt editing. Drawing inspiration from the actor-critic algorithm in reinforcement learning, PACE leverages LLMs as the dual roles of actors and critics, conceptualizing prompt as a type of policy. PACE refines prompt, taking into account the feedback from both actors performing prompt and critics criticizing response. This process helps LLMs better align prompt to a specific task, thanks to real responses and thinking from LLMs. We conduct extensive experiments on 24 instruction induction tasks and 21 big-bench tasks. Experimental results indicate that PACE elevates the relative performance of medium/low-quality human-written prompts by up to 98%, which has comparable performance to high-quality human-written prompts. Moreover, PACE also exhibits notable efficacy for prompt generation.