mlsanityData to 7 Oct 2026

Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs

arXiv 2406.11695 ↗First posted 17 Jun 2024cs.CL cs.AI cs.LGcited by 270

Not on the map. Why?

Cited by 93 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Language Model Programs, i.e. sophisticated pipelines of modular language model (LM) calls, are increasingly advancing NLP tasks, but they require crafting prompts that are jointly effective for all modules. We study prompt optimization for LM programs, i.e. how to update these prompts to maximize a downstream metric without access to module-level labels or gradients. To make this tractable, we factorize our problem into optimizing the free-form instructions and few-shot demonstrations of every module and introduce several strategies to craft task-grounded instructions and navigate credit assignment across modules. Our strategies include (i) program- and data-aware techniques for proposing effective instructions, (ii) a stochastic mini-batch evaluation function for learning a surrogate model of our objective, and (iii) a meta-optimization procedure in which we refine how LMs construct proposals over time. Using these insights we develop MIPRO, a novel algorithm for optimizing LM programs. MIPRO outperforms baseline optimizers on five of seven diverse multi-stage LM programs using a best-in-class open-source model (Llama-3-8B), by as high as 13% accuracy. We have released our new optimizers and benchmark in DSPy at http://dspy.ai

Key facts. Cited by 270 papers in the map (data through 7 Oct 2026). Builds on 26 of its 52 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

26 of 52 references.

References · 26

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  3. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  5. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  6. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  7. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  8. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  9. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  10. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  11. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
  12. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 521
14 more
  1. 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 466
  2. 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 465
  3. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  4. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
  5. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 316
  6. 2021AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Promptscited by 265
  7. 2020HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verificationcited by 199
  8. 2023EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizerscited by 189
  9. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 136
  10. 2022Language Model Cascadescited by 104
  11. 2024Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineeringcited by 100
  12. 2021Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrievalcited by 63
  13. 2023Joint Prompt Optimization of Stacked LLMs using Variational Inferencecited by 32
  14. 2023ScoNe: Benchmarking Negation Reasoning in Language Models With Fine-Tuning and In-Context Learningcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 79; 2026: 181.

270 papers.

Sort:

By area:

  1. 2025Agentic Context Engineering: Evolving Contexts for Self-Improving Language Modelscited by 349
  2. 2026Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnessescited by 119
  3. 2025Multi-Agent Design: Optimizing Agents with Better Prompts and Topologiescited by 106
  4. 2024PAPILLON: Privacy Preservation from Internet-based and Local Language Model Ensemblescited by 54
  5. 2025EvoAgentX: An Automated Framework for Evolving Agentic Workflowscited by 43
  6. 2025Feedback Descent: Open-Ended Text Optimization via Pairwise Comparisoncited by 42
  7. 2026Meta Context Engineering via Agentic Skill Evolutioncited by 42
  8. 2024Fine-Tuning and Prompt Optimization: Two Great Steps that Work Better Togethercited by 38
  9. 2026HarnessX: A Composable, Adaptive, and Evolvable Agent Harness Foundrycited by 37
  10. 2026Continual Harness: Online Adaptation for Self-Improving Foundation Agentscited by 30
  11. 2025LLM Agents Should Employ Security Principlescited by 28
  12. 2025SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMscited by 26
38 more
  1. 2025Optimas: Optimizing Compound AI Systems with Globally Aligned Local Rewardscited by 23
  2. 2024StructuredRAG: JSON Response Formatting with Large Language Modelscited by 22
  3. 2025From Few to Many: Self-Improving Many-Shot Reasoners Through Iterative Optimization and Generationcited by 20
  4. 2025How to Train Your Advisor: Steering Black-Box LLMs with Advisor Modelscited by 20
  5. 2025Searching for Privacy Risks in LLM Agents via Simulationcited by 19
  6. 2025SEW: Self-Evolving Agentic Workflows for Automated Code Generationcited by 18
  7. 2024Grounding by Trying: LLMs with Reinforcement Learning-Enhanced Retrievalcited by 17
  8. 2025Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Taskscited by 17
  9. 2025Improved Representation Steering for Language Modelscited by 17
  10. 2024Iteration of Thought: Leveraging Inner Dialogue for Autonomous Large Language Model Reasoningcited by 16
  11. 2025Adaptive Self-improvement LLM Agentic System for ML Library Developmentcited by 16
  12. 2026Co-Harness: Co-Evolving Harnesses and Model Weights for LLM Agentscited by 16
  13. 2025MAPRO: Recasting Multi-Agent Prompt Optimization as Maximum a Posteriori Inferencecited by 15
  14. 2026Model Context Protocol (MCP) Tool Descriptions Are Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptionscited by 15
  15. 2025CORRECT: COndensed eRror RECognition via knowledge Transfer in multi-agent systemscited by 14
  16. 2026MemoHarness: Agent Harnesses That Learn from Experiencecited by 14
  17. 2026Harness-R1: Learning to Edit Executable Runtime Harnesses from Agent Failure Trajectoriescited by 13
  18. 2025Abacus: A Cost-Based Optimizer for Semantic Operator Systemscited by 11
  19. 2026Synthesizing Multi-Agent Harnesses for Vulnerability Discoverycited by 11
  20. 2025GAAPO: Genetic Algorithmic Applied to Prompt Optimizationcited by 10
  21. 2026KernelBlaster: Continual Cross-Task CUDA Optimization via Memory-Augmented In-Context Reinforcement Learningcited by 10
  22. 2024Neuro-Symbolic Evaluation of Text-to-Video Models using Formal Verificationcited by 9
  23. 2024Hyperband-based Bayesian Optimization for Black-box Prompt Selectioncited by 9
  24. 2025AegisLLM: Scaling Agentic Systems for Self-Reflective Defense in LLM Securitycited by 9
  25. 2025PRL: Prompts from Reinforcement Learningcited by 9
  26. 2025Maestro: Joint Graph & Config Optimization for Reliable AI Agentscited by 9
  27. 2026AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Tracescited by 9
  28. 2025EnronQA: Towards Personalized RAG over Private Documentscited by 8
  29. 2025LLM Prompt Duel Optimizer: Efficient Label-Free Prompt Optimizationcited by 8
  30. 2026ACM: Agentic Context Management for Long Horizon Taskscited by 8
  31. 2024A Comparative Study of DSPy Teleprompter Algorithms for Aligning Large Language Models Evaluation Metrics to Human Evaluationcited by 7
  32. 2025Meta-Prompt Optimization for LLM-Based Sequential Decision Makingcited by 7
  33. 2025Cognify: Supercharging Gen-AI Workflows With Hierarchical Autotuningcited by 7
  34. 2025AutoMedPrompt: A New Framework for Optimizing LLM Medical Prompts Using Textual Gradientscited by 7
  35. 2025FEM-Bench: A Structured Scientific Reasoning Benchmark for Evaluating Code-Generating LLMscited by 7
  36. 2026Constitutional Black-Box Monitoring for Scheming in LLM Agentscited by 7
  37. 2026AI Research Preference Modelscited by 7
  38. 2026Verify Smarter, Evolve Further: Efficient Harness Evolution through Behavior-Aware Verificationcited by 7

50 most cited of 270 All 270 (JSON)