mlsanityData to 7 Oct 2026

Why Prompt Optimization Works, and Why It Sometimes Doesn't: A Causal-Inspired Edit-Level Analysis

arXiv 2605.26655 ↗First posted 26 May 2026cs.CL cs.LG cs.NEcited by 0

Not on the map. Why?

Abstract (arXiv)

Automated prompt optimization methods (e.g., DSpy, TextGrad) can substantially improve the performance of large language model (LLM), however, their generalization ability across different tasks remains underperformed. In practice, the superiority of the optimized prompt on one benchmark often fails to transfer to another, and this limitation persists even when switching across different LLM backbones. To investigate the underexplored sources of heterogeneity in prompt performance, we conduct a causal inference-inspired observational analysis of optimized prompts across a diverse set of optimization frameworks, LLM backbones, and NLP benchmarks. To achieve the goal, we build upon the propensity-adjusted associational analysis together with multiple complementary representations of prompt edits, where the consistent task-conditioned edits patterns are identified. We find that complexity-increasing and meta-instructional edits are negatively associated with mathematical and multi-hop reasoning performance, whereas step-by-step and meta-cognitive edits improve logical and sequential reasoning tasks. These effects are robust across cognitive-load annotations, surface-level text features, and edit-motif analyses, and can generalize across optimization frameworks. Overall, these results indicate that prompt optimization failures arise from systematic interactions between edit families and task characteristics rather than random optimization artifacts, providing feature-level characterization of optimizer behavior and motivating future task-conditioned optimizer design.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 33 of its 48 references that are in the map. First posted on arXiv 26 May 2026.

Builds on

33 of 48 references.

References · 33

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  2. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  3. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  4. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  5. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  6. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  7. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  8. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  9. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  10. 2023Large Language Models as Optimizerscited by 830
  11. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  12. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
21 more
  1. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
  2. 2015Estimation and Inference of Heterogeneous Treatment Effects using Random Forestscited by 597
  3. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 521
  4. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  5. 2016Solving General Arithmetic Word Problemscited by 353
  6. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  7. 2019Measuring Compositional Generalization: A Comprehensive Method on Realistic Datacited by 280
  8. 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 270
  9. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  10. 2023Can Large Language Models Infer Causation from Correlation?cited by 142
  11. 2023Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactualscited by 44
  12. 2024Causal Inference with Large Language Model: A Surveycited by 31
  13. 2024End-To-End Causal Effect Estimation from Unstructured Natural Language Datacited by 18
  14. 2025Compositional Causal Reasoning Evaluation in Language Modelscited by 6
  15. 2025Automatic Prompt Optimization for Knowledge Graph Construction: Insights from an Empirical Studycited by 6
  16. 2026Optimizing Prompts for Large Language Models: A Causal Approachcited by 3
  17. 2026Multi-Sourced, Multi-Agent Evidence Retrieval for Fact-Checkingcited by 3
  18. 2026Prompt Optimization Is a Coin Flip: Diagnosing When It Helps in Compound AI Systemscited by 3
  19. 2026Self-Optimizing Multi-Agent Systems for Deep Researchcited by 2
  20. 2026TextReg: Mitigating Prompt Distributional Overfitting via Regularized Text-Space Optimizationcited by 2
  21. 2026A Reproducible Optimisation Protocol for Calibrating Prompt-Based Large Language Model Workflows in Evidence Synthesiscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Eliciting Causal Abilities in Large Language Models for Reasoning Tasks
  2. 2024GReaTer: Gradients over Reasoning Makes Smaller Language Models Strong Prompt Optimizers
  3. 2025Self-Supervised Prompt Optimization
  4. 2025How to Auto-optimize Prompts for Domain Tasks? Adaptive Prompting and Reasoning through Evolutionary Domain Knowledge Adaptation
  5. 2026Modular Prompt Optimization: Optimizing Structured Prompts with Section-Local Textual Gradients
  6. 2025Better by Comparison: Retrieval-Augmented Contrastive Reasoning for Automatic Prompt Optimization
  7. 2025Cross-Lingual Prompt Steerability: Towards Accurate and Robust LLM Behavior across Languages
  8. 2026How Much Prompt Is Enough? A Blackbox Minimization of Few-Shots in LLMs