# The Unreasonable Effectiveness of Eccentric Automatic Prompts

arXiv: https://arxiv.org/abs/2402.10949 · HTML: https://mlsanity.com/paper/2402.10949/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 9 of its 9 references that are in the map.
- First posted on arXiv 9 Feb 2024.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)

## Builds on (most cited first, 9 of 9)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)

## Built on by (most cited first, 12 of 12)

- [Large Language Models Miss the Multi-Agent Mark](https://mlsanity.com/paper/2505.21298/) (2025)
- [Token embeddings violate the manifold hypothesis](https://mlsanity.com/paper/2504.01002/) (2025)
- [LASSI: An LLM-based Automated Self-Correcting Pipeline for Translating Parallel Scientific Codes](https://mlsanity.com/paper/2407.01638/) (2024)
- [Why is prompting hard? Understanding prompts on binary sequence predictors](https://mlsanity.com/paper/2502.10760/) (2025)
- [Sequential Enumeration in Large Language Models](https://mlsanity.com/paper/2512.04727/) (2025)
- [Prompt Selection Matters: Enhancing Text Annotations for Social Sciences with Large Language Models](https://mlsanity.com/paper/2407.10645/) (2024)
- [Critical Confabulation: Can LLMs Hallucinate for Social Good?](https://mlsanity.com/paper/2511.07722/) (2025)
- [More Is Not Always Better: Cross-Component Interference in LLM Agent Scaffolding](https://mlsanity.com/paper/2605.05716/) (2026)
- [SandboxAQ's submission to MRL 2024 Shared Task on Multi-lingual Multi-task Information Retrieval](https://mlsanity.com/paper/2410.21501/) (2024)
- [AI-Facilitated Analysis of Abstracts and Conclusions: Flagging Unsubstantiated Claims and Ambiguous Pronouns](https://mlsanity.com/paper/2506.13172/) (2025)
- [Empirical Prompt Engineering for Construct Identification with Large Language Models](https://mlsanity.com/paper/2512.03818/) (2025)
- [PerMix-RLVR: Preserving Persona Expressivity under Verifiable-Reward Alignment](https://mlsanity.com/paper/2604.08986/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable problem-solving and basic mathematics abilities. However, their efficacy is highly contingent on the formulation of the prompt. This study endeavors to quantify the influence of incorporating "positive thinking" into the system message of the prompt, then compare that to systematic prompt optimization. We assess the performance of 60 combinations of system message snippets, tested with and without Chain of Thought prompting, across three models with parameters ranging from 7 to 70 billion on the GSM8K dataset. Our findings reveal that results do not universally generalize across models. In most instances, the inclusion of "positive thinking" prompts positively affected model performance. Notably, however, Llama2-70B exhibited an exception when not utilizing Chain of Thought, as the optimal system message was found to be none at all. Given the combinatorial complexity, and thus computation time, of experimenting with hand-tuning prompts for large black-box models, we then compared the performance of the best "positive thinking" prompt against the output of systematic prompt optimization. We show that employing an automated prompt optimizer emerges as the most effective method for enhancing performance, even when working with smaller open-source models. Additionally, our findings reveal that the highest-scoring, automatically-optimized prompt exhibits a degree of peculiarity far beyond expectations.