mlsanityData to 14 Oct 2026

Beyond Aggregate Accuracy: Item-Level Transitions in Prompt Optimization

arXiv 2610.19084 ↗First posted 13 Oct 2026cs.LG cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Automatic prompt optimization (APO) uses feedback from model responses to improve prompts automatically. However, its effectiveness is usually measured through overall scores, which do not reveal which questions become correct, which previously correct answers become wrong, or whether improvements extend to unseen questions. We introduce Reflective Feedback-Driven Prompt Optimization (RFDPO), a tool that tracks recoveries (incorrect to correct), regressions (correct to incorrect), and churn (all changed outcomes). RFDPO uses this feedback to guide subsequent revisions and records prompt changes throughout optimization. We evaluate RFDPO across five benchmarks covering law, knowledge, instruction following, mathematics, and privacy, using GPT-5 as the optimizer and GPT-4o-mini and GPT-4.1 Mini as solvers. Our results show that overall accuracy can hide substantial changes: in five of nine configurations, churn was at least five times the absolute net change, and a single-item net change concealed up to 17 changed outcomes. Average validation gains were 7.9 percentage points, compared with 5.2 points on held-out tests. In a controlled ablation on LegalBench-Hearsay, item-level feedback reduced churn by 40% (p=0.024) while maintaining similar net gains. These findings motivate reporting recovery, regression, churn, and performance on unseen data alongside overall accuracy.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 35 references that are in the map. First posted on arXiv 13 Oct 2026.

Builds on

22 of 35 references.

References · 22

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  3. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  4. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  5. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  6. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  7. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  8. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 653
  9. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 551
  10. 2025GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningcited by 542
  11. 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 291
  12. 2023Proving Test Set Contamination in Black Box Language Modelscited by 230
10 more
  1. 2025Generalizing Verifiable Instruction Followingcited by 194
  2. 2024The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect Large Language Model Performancecited by 100
  3. 2024What Did I Do Wrong? Quantifying LLMs' Sensitivity and Consistency to Prompt Engineeringcited by 90
  4. 2020Positive-Congruent Training: Towards Regression-Free Model Updatescited by 57
  5. 2024PAPILLON: Privacy Preservation from Internet-based and Local Language Model Ensemblescited by 57
  6. 2025Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMscited by 30
  7. 2023(Why) Is My Prompt Getting Worse? Rethinking Regression Testing for Evolving LLM APIscited by 23
  8. 2026Modular Prompt Optimization: Optimizing Structured Prompts with Section-Local Textual Gradientscited by 8
  9. 2025Knowing How to Edit: Reliable Evaluation Signals for Diagnosing and Optimizing Prompts at Query Levelcited by 2
  10. 2026Trace2Policy: From Expert Behavior Traces to Self-Evolving Decision Agentscited by 1

Built on by

Not cited yet.