# Revisiting OPRO: The Limitations of Small-Scale LLMs as Optimizers

arXiv: https://arxiv.org/abs/2405.10276 · HTML: https://mlsanity.com/paper/2405.10276/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 18 references that are in the map.
- First posted on arXiv 16 May 2024.

## Builds on (most cited first, 13 of 13)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [Large Language Models Cannot Self-Correct Reasoning Yet](https://mlsanity.com/paper/2310.01798/) (2023)
- [Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://mlsanity.com/paper/2305.02301/) (2023)
- [GPT Understands, Too](https://mlsanity.com/paper/2103.10385/) (2021)
- [Automatic Prompt Optimization with "Gradient Descent" and Beam Search](https://mlsanity.com/paper/2305.03495/) (2023)
- [Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution](https://mlsanity.com/paper/2309.16797/) (2023)
- [EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizers](https://mlsanity.com/paper/2309.08532/) (2023)
- [Efficient Large Language Models: A Survey](https://mlsanity.com/paper/2312.03863/) (2023)
- [Revisiting Automated Prompting: Are We Actually Doing Better?](https://mlsanity.com/paper/2304.03609/) (2023)

## Built on by (most cited first, 15 of 15)

- [Aviary: training language agents on challenging scientific tasks](https://mlsanity.com/paper/2412.21154/) (2024)
- [PromptSculptor: Multi-Agent Based Text-to-Image Prompt Optimization](https://mlsanity.com/paper/2509.12446/) (2025)
- [GReaTer: Gradients over Reasoning Makes Smaller Language Models Strong Prompt Optimizers](https://mlsanity.com/paper/2412.09722/) (2024)
- [A Systematic Survey on Large Language Models for Evolutionary Optimization: From Modeling to Solving](https://mlsanity.com/paper/2509.08269/) (2025)
- [Robust Utility-Preserving Text Anonymization Based on Large Language Models](https://mlsanity.com/paper/2407.11770/) (2024)
- [When Is an LLM Worth It for Hyperparameter Optimization? A Budget-Matched Study on Tabular Data Finds the Warm-Start Is a Default Configuration, Not the Model](https://mlsanity.com/paper/2606.21641/) (2026)
- [PMPO: Probabilistic Metric Prompt Optimization for Small and Large Language Models](https://mlsanity.com/paper/2505.16307/) (2025)
- [Bounds of Chain-of-Thought Robustness: Reasoning Steps, Embed Norms, and Beyond](https://mlsanity.com/paper/2509.21284/) (2025)
- [GFlowPO: Generative Flow Network as a Language Model Prompt Optimizer](https://mlsanity.com/paper/2602.03358/) (2026)
- [Improving Multi-turn Dialogue Consistency with Self-Recall Thinking](https://mlsanity.com/paper/2605.15102/) (2026)
- [LM-GRASP: Instance-Specific Language Models for Combinatorial Construction via Online Imitation Learning](https://mlsanity.com/paper/2607.28135/) (2026)
- [From Monolithic to Modular: Segment-level Automatic Prompt Optimization](https://mlsanity.com/paper/2608.11219/) (2026)
- [CHART: A Harness-Rotation Curriculum for Harness-Robust Search Agents](https://mlsanity.com/paper/2609.22247/) (2026)
- [Teach-to-Crash: A Closed-Loop Student-Teacher LLM Framework for Collision-Inducing Test Scenario Generation](https://mlsanity.com/paper/2609.27296/) (2026)
- [Visible but Not Avoided: State-Dependent Re-Proposal of Rejected Candidates in Catalog-Based LLM Optimization Loops](https://mlsanity.com/paper/2610.15738/) (2026)

## Abstract (arXiv)

Numerous recent works aim to enhance the efficacy of Large Language Models (LLMs) through strategic prompting. In particular, the Optimization by PROmpting (OPRO) approach provides state-of-the-art performance by leveraging LLMs as optimizers where the optimization task is to find instructions that maximize the task accuracy. In this paper, we revisit OPRO for automated prompting with relatively small-scale LLMs, such as LLaMa-2 family and Mistral 7B. Our investigation reveals that OPRO shows limited effectiveness in small-scale LLMs, with limited inference capabilities constraining optimization ability. We suggest future automatic prompting engineering to consider both model capabilities and computational costs. Additionally, for small-scale LLMs, we recommend direct instructions that clearly outline objectives and methodologies as robust prompt baselines, ensuring efficient and effective prompt engineering in ongoing research.