mlsanityData to 14 Oct 2026

Optimization before Evaluation: Evaluation with Unoptimised Prompts Can be Misleading

arXiv 2604.27637 ↗First posted 30 Apr 2026cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Current Large Language Model (LLM) evaluation frameworks utilize the same static prompt template across all models under evaluation. This differs from the common industry practice of using prompt optimization (PO) techniques to optimize the prompt for each model to maximize application performance. In this paper, we investigate the effect of PO towards LLM evaluations. Our results on public academic and internal industry benchmarks show that PO greatly affects the final ranking of models. This highlights the importance of practitioners performing PO per model when conducting evaluations to choose the best model for a given task.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 16 of its 40 references that are in the map. First posted on arXiv 30 Apr 2026.

Builds on

16 of 40 references.

References · 16

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  5. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  6. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  7. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 653
  8. 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 291
  9. 2023Black-Box Prompt Optimization: Aligning Large Language Models without Model Trainingcited by 122
  10. 2023From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuningcited by 56
  11. 2021RAFT: A Real-World Few-Shot Text Classification Benchmarkcited by 44
  12. 2024SPRIG: Improving Large Language Model Performance by System Prompt Optimizationcited by 29
4 more
  1. 2023LLMeBench: A Flexible Framework for Accelerating LLMs Benchmarkingcited by 20
  2. 2024Unitxt: Flexible, Shareable and Reusable Data Preparation and Evaluation for Generative AIcited by 18
  3. 2025Beyond Prompt Content: Enhancing LLM Performance via Content-Format Integrated Prompt Optimizationcited by 16
  4. 2024FIPO: Free-form Instruction-oriented Prompt Optimization with Preference Dataset and Modular Fine-tuning Schemacited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026The Wording Effect: Quantifying Two-Way Drift in LLM Benchmark Performancecited by 1