mlsanityData to 7 Oct 2026

DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines

arXiv 2310.03714 ↗First posted 5 Oct 2023cs.CL cs.AI cs.IRcited by 803

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

The ML community is rapidly exploring techniques for prompting language models (LMs) and for stacking them into pipelines that solve complex tasks. Unfortunately, existing LM pipelines are typically implemented using hard-coded "prompt templates", i.e. lengthy strings discovered via trial and error. Toward a more systematic approach for developing and optimizing LM pipelines, we introduce DSPy, a programming model that abstracts LM pipelines as text transformation graphs, i.e. imperative computational graphs where LMs are invoked through declarative modules. DSPy modules are parameterized, meaning they can learn (by creating and collecting demonstrations) how to apply compositions of prompting, finetuning, augmentation, and reasoning techniques. We design a compiler that will optimize any DSPy pipeline to maximize a given metric. We conduct two case studies, showing that succinct DSPy programs can express and optimize sophisticated LM pipelines that reason about math word problems, tackle multi-hop retrieval, answer complex questions, and control agent loops. Within minutes of compiling, a few lines of DSPy allow GPT-3.5 and llama2-13b-chat to self-bootstrap pipelines that outperform standard few-shot prompting (generally by over 25% and 65%, respectively) and pipelines with expert-created demonstrations (by up to 5-46% and 16-40%, respectively). On top of that, DSPy programs compiled to open and relatively small LMs like 770M-parameter T5 and llama2-13b-chat are competitive with approaches that rely on expert-written prompt chains for proprietary GPT-3.5. DSPy is available at https://github.com/stanfordnlp/dspy

Key facts. Cited by 803 papers in the map (data through 7 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 56 of its 112 references that are in the map. First posted on arXiv 5 Oct 2023.

Builds on

56 of 112 references.

References · 56

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  8. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  9. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  12. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
38 more
  1. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  2. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  3. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  4. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  5. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  6. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  7. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  8. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  9. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  10. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  11. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  12. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  13. 2023Large Language Models as Optimizerscited by 830
  14. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  15. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  16. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  17. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  18. 2022Large Language Models Can Self-Improvecited by 656
  19. 2022PAL: Program-aided Language Modelscited by 635
  20. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
  21. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  22. 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 465
  23. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  24. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 316
  25. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 286
  26. 2022Prompting GPT-3 To Be Reliablecited by 258
  27. 2016Data Programming: Creating Large Training Sets, Quicklycited by 246
  28. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 235
  29. 2023EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizerscited by 189
  30. 2022MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoningcited by 153
  31. 2022Internet-augmented language models through few-shot prompting for open-domain question answeringcited by 131
  32. 2022Rationale-Augmented Ensembles in Language Modelscited by 111
  33. 2022Prompting Is Programming: A Query Language for Large Language Modelscited by 110
  34. 2022Language Model Cascadescited by 104
  35. 2023Answering Questions by Meta-Reasoning over Multiple Chains of Thoughtcited by 87
  36. 2020Relevance-guided Supervision for OpenQA with ColBERTcited by 71
  37. 2022Recitation-Augmented Language Modelscited by 65
  38. 2016Theano: A Python framework for fast computation of mathematical expressionscited by 64

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 132; 2025: 231; 2026: 432.

803 papers.

Sort:

By area:

  1. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  2. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  3. 2025Why Do Multi-Agent LLM Systems Fail?cited by 559
  4. 2025Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agentscited by 294
  5. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 292
  6. 2026SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Taskscited by 289
  7. 2026Meta-Harness: End-to-End Optimization of Model Harnessescited by 248
  8. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 223
  9. 2025From RAG to Memory: Non-Parametric Continual Learning for Large Language Modelscited by 223
  10. 2024Data Interpreter: An LLM Agent For Data Sciencecited by 222
  11. 2024AI Agents That Mattercited by 199
  12. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
38 more
  1. 2025Multi-agent Architecture Search via Agentic Supernetcited by 164
  2. 2024Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generationcited by 158
  3. 2024LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Aheadcited by 155
  4. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  5. 2024Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Modelscited by 150
  6. 2024Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferencescited by 142
  7. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  8. 2024Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospectscited by 130
  9. 2024Cut the Crap: An Economical Communication Pipeline for LLM-based Multi-Agent Systemscited by 129
  10. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  11. 2026SkillOpt: Executive Strategy for Self-Evolving Agent Skillscited by 122
  12. 2026Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnessescited by 119
  13. 2023AutoMix: Automatically Mixing Language Modelscited by 116
  14. 2024Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffoldingcited by 115
  15. 2024G-Designer: Architecting Multi-agent Communication Topologies via Graph Neural Networkscited by 110
  16. 2025AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?cited by 105
  17. 2024Symbolic Learning Enables Self-Evolving Agentscited by 104
  18. 2024InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationalescited by 100
  19. 2025S*: Test Time Scaling for Code Generationcited by 95
  20. 2025Agent KB: Leveraging Cross-Domain Experience for Agentic Problem Solvingcited by 90
  21. 2025AgentNet: Decentralized Evolutionary Coordination for LLM-based Multi-Agent Systemscited by 86
  22. 2024Language Agents as Optimizable Graphscited by 81
  23. 2023ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agentcited by 78
  24. 2024AutoFlow: Automated Workflow Generation for Large Language Model Agentscited by 76
  25. 2024Trace is the Next AutoDiff: Generative Optimization with Rich Feedback, Execution Traces, and LLMscited by 62
  26. 2024Archon: An Architecture Search Framework for Inference-Time Techniquescited by 59
  27. 2024Tool Learning in the Wild: Empowering Language Models as Automatic Tool Agentscited by 56
  28. 2025ThetaEvolve: Test-time Learning on Open Problemscited by 52
  29. 2024Multi-step Problem Solving Through a Verifier: An Empirical Analysis on Model-induced Process Supervisioncited by 51
  30. 2024Generate-then-Ground in Retrieval-Augmented Generation for Multi-hop Question Answeringcited by 51
  31. 2025MARFT: Multi-Agent Reinforcement Fine-Tuningcited by 51
  32. 2025EvoFlow: Evolving Diverse Agentic Workflows On The Flycited by 49
  33. 2024Project Sid: Many-agent simulations toward AI civilizationcited by 46
  34. 2025TaskCraft: Automated Generation of Agentic Taskscited by 46
  35. 2025In-depth Analysis of Graph-based RAG in a Unified Frameworkcited by 45
  36. 2024Into the Unknown Unknowns: Engaged Human Learning through Participation in Language Model Agent Conversationscited by 43
  37. 2025EvoAgentX: An Automated Framework for Evolving Agentic Workflowscited by 43
  38. 2026SkillForge: Forging Domain-Specific, Self-Evolving Agent Skills in Cloud Technical Supportcited by 43

50 most cited of 803 All 803 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Optimizing LLM Prompt Engineering with DSPy Based Declarative Learning
  2. 2024Towards Generalist Prompting for Large Language Models by Mental Models
  3. 2025Probing Large Language Models in Reasoning and Translating Complex Linguistic Puzzles
  4. 2025Recursive Language Models
  5. 2026Evaluating Prompting and Execution-Based Methods for Deterministic Computation in LLMs
  6. 2025PromptCoT 2.0: Scaling Prompt Synthesis for Large Language Model Reasoning
  7. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Models
  8. 2024Demystifying Chains, Trees, and Graphs of Thoughts