# Evoke: Evoking Critical Thinking Abilities in LLMs via Reviewer-Author Prompt Editing

arXiv: https://arxiv.org/abs/2310.13855 · HTML: https://mlsanity.com/paper/2310.13855/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 29 of its 33 references that are in the map.
- First posted on arXiv 20 Oct 2023.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)

## Builds on (most cited first, 29 of 29)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [PAL: Program-aided Language Models](https://mlsanity.com/paper/2211.10435/) (2022)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://mlsanity.com/paper/2102.07350/) (2021)
- [Towards Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2212.10403/) (2022)
- [Prompting Large Language Model for Machine Translation: A Case Study](https://mlsanity.com/paper/2301.07069/) (2023)
- [On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective](https://mlsanity.com/paper/2302.12095/) (2023)
- [Guiding Pretraining in Reinforcement Learning with Large Language Models](https://mlsanity.com/paper/2302.06692/) (2023)
- [Instruction Induction: From Few Examples to Natural Language Task Descriptions](https://mlsanity.com/paper/2205.10782/) (2022)
- [Automatic Generation of Programming Exercises and Code Explanations using Large Language Models](https://mlsanity.com/paper/2206.11861/) (2022)
- [Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations](https://mlsanity.com/paper/2205.12685/) (2022)
- [Repository-Level Prompt Generation for Large Language Models of Code](https://mlsanity.com/paper/2206.12839/) (2022)
- [Robust Encodings: A Framework for Combating Adversarial Typos](https://mlsanity.com/paper/2005.01229/) (2020)
- [DeepTagger: Knowledge Enhanced Named Entity Recognition for Web-Based Ads Queries](https://mlsanity.com/paper/2306.17413/) (2023)

## Built on by (most cited first, 11 of 11)

- [Prompt Engineering a Prompt Engineer](https://mlsanity.com/paper/2311.05661/) (2023)
- [Task Facet Learning: A Structured Approach to Prompt Optimization](https://mlsanity.com/paper/2406.10504/) (2024)
- [Efficient and Accurate Prompt Optimization: the Benefit of Memory in Exemplar-Guided Reflection](https://mlsanity.com/paper/2411.07446/) (2024)
- [StraGo: Harnessing Strategic Guidance for Prompt Optimization](https://mlsanity.com/paper/2410.08601/) (2024)
- [Memory in Large Language Models: Mechanisms, Evaluation and Evolution](https://mlsanity.com/paper/2509.18868/) (2025)
- [Competence-Aware AI Agents with Metacognition for Unknown Situations and Environments (MUSE)](https://mlsanity.com/paper/2411.13537/) (2024)
- [Concept Distillation from Strong to Weak Models via Hypotheses-to-Theories Prompting](https://mlsanity.com/paper/2408.09365/) (2024)
- [The Model Agreed, But Didn't Learn: Diagnosing Surface Compliance in Large Language Models](https://mlsanity.com/paper/2604.05995/) (2026)
- [ELPO: Ensemble Learning Based Prompt Optimization for Large Language Models](https://mlsanity.com/paper/2511.16122/) (2025)
- [PromptTailor: Multi-turn Intent-Aligned Prompt Synthesis for Lightweight LLMs](https://mlsanity.com/paper/2511.21725/) (2025)
- [A Dataset for Dynamic Human Preferences for Vision Language Models](https://mlsanity.com/paper/2606.07653/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have made impressive progress in natural language processing. These models rely on proper human instructions (or prompts) to generate suitable responses. However, the potential of LLMs are not fully harnessed by commonly-used prompting methods: many human-in-the-loop algorithms employ ad-hoc procedures for prompt selection; while auto prompt generation approaches are essentially searching all possible prompts randomly and inefficiently. We propose Evoke, an automatic prompt refinement framework. In Evoke, there are two instances of a same LLM: one as a reviewer (LLM-Reviewer), it scores the current prompt; the other as an author (LLM-Author), it edits the prompt by considering the edit history and the reviewer's feedback. Such an author-reviewer feedback loop ensures that the prompt is refined in each iteration. We further aggregate a data selection approach to Evoke, where only the hard samples are exposed to the LLM. The hard samples are more important because the LLM can develop deeper understanding of the tasks out of them, while the model may already know how to solve the easier cases. Experimental results show that Evoke significantly outperforms existing methods. For instance, in the challenging task of logical fallacy detection, Evoke scores above 80, while all other baseline methods struggle to reach 20.