mlsanityData to 14 Oct 2026

Evolutionary Pre-Prompt Optimization for Mathematical Reasoning

arXiv 2412.04291 ↗First posted 5 Dec 2024cs.CLcited by 2

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Recent advancements have highlighted that large language models (LLMs), when given a small set of task-specific examples, demonstrate remarkable proficiency, a capability that extends to complex reasoning tasks. In particular, the combination of few-shot learning with the chain-of-thought (CoT) approach has been pivotal in steering models towards more logically consistent conclusions [Wei et al. 2022b]. This paper explores the optimization of example selection for designing effective CoT pre-prompts and shows that the choice of the optimization algorithm, typically in favor of comparison-based methods such as evolutionary computation, significantly enhances efficacy and feasibility. Specifically, thanks to a limited exploitative and overfitted optimization, Evolutionary Pre-Prompt Optimization (EPPO) brings an improvement over the naive few-shot approach, exceeding 10 absolute points in exact match scores on benchmark datasets such as GSM8k and MathQA. These gains are consistent across various contexts and are further amplified when integrated with self-consistency (SC).

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 28 of its 52 references that are in the map. First posted on arXiv 5 Dec 2024.

Builds on

28 of 52 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Fewer is More: Boosting LLM Reasoning with Reinforced Context Pruning14 Dec
  2. 2023Resprompt: Residual Connection Prompting Advances Multi-Step Reasoning in Large Language Models7 Oct
  3. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them17 Oct
  4. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Models21 May
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar

Other references · 23

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  5. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  6. 2022Emergent Abilities of Large Language Modelscited by 2,512
  7. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  8. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  9. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  10. 2023PaLM 2 Technical Reportcited by 1,162
  11. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  12. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
11 more
  1. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 872
  2. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  3. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  4. 2024Iterative Reasoning Preference Optimizationcited by 246
  5. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 161
  6. 2024BOND: Aligning LLMs with Best-of-N Distillationcited by 76
  7. 2023TinyGSM: achieving >80% on GSM8k with small language modelscited by 70
  8. 2023OpenICL: An Open-Source Framework for In-context Learningcited by 51
  9. 2024What Makes Large Language Models Reason in (Multi-Turn) Code Generation?cited by 29
  10. 2023IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Modelscited by 27
  11. 2024Evolutionary Retrofittingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Tuning without Peeking: Provable Generalization Bounds and Robust LLM Post-Trainingcited by 1
  2. 2026Evolving LLM-Generated Features for Interpretable Classificationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Zero-Shot Chain-of-Thought Reasoning Guided by Evolutionary Algorithms in Large Language Models
  2. 2022Large Language Models are Zero-Shot Reasoners
  3. 2023An automatically discovered chain-of-thought prompt generalizes to novel models and datasets
  4. 2025PREMISE: Scalable and Strategic Prompt Optimization for Efficient Mathematical Reasoning in Large Models
  5. 2026CoTEvol: Self-Evolving Chain-of-Thoughts for Data Synthesis in Mathematical Reasoning
  6. 2023Leveraging Training Data in Few-Shot Prompting for Numerical Reasoning
  7. 2026Soft Guidance Starts to Outperform CoT Prompting as LLMs Improve
  8. 2025Evaluating GRPO and DPO for Faithful Chain-of-Thought Reasoning in LLMs