# DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines

arXiv: https://arxiv.org/abs/2312.13382 · HTML: https://mlsanity.com/paper/2312.13382/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 38 references that are in the map.
- First posted on arXiv 20 Dec 2023.

## Builds on (most cited first, 18 of 18)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions](https://mlsanity.com/paper/2212.10509/) (2022)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [Enabling Large Language Models to Generate Text with Citations](https://mlsanity.com/paper/2305.14627/) (2023)
- [ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction](https://mlsanity.com/paper/2112.01488/) (2021)
- [Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP](https://mlsanity.com/paper/2212.14024/) (2022)
- [Lexically Constrained Decoding for Sequence Generation Using Grid Beam Search](https://mlsanity.com/paper/1704.07138/) (2017)
- [Prompting Is Programming: A Query Language for Large Language Models](https://mlsanity.com/paper/2212.06094/) (2022)
- [Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement](https://mlsanity.com/paper/2310.08559/) (2023)
- [LLMs cannot find reasoning errors, but can correct them given the error location](https://mlsanity.com/paper/2311.08516/) (2023)
- [TrueTeacher: Learning Factual Consistency Evaluation with Large Language Models](https://mlsanity.com/paper/2305.11171/) (2023)
- [Automatic Evaluation of Attribution by Large Language Models](https://mlsanity.com/paper/2305.06311/) (2023)
- [Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrieval](https://mlsanity.com/paper/2101.00436/) (2021)
- [LLMRefine: Pinpointing and Refining Large Language Models via Fine-Grained Actionable Feedback](https://mlsanity.com/paper/2311.09336/) (2023)
- [Model Assertions for Monitoring and Improving ML Models](https://mlsanity.com/paper/2003.01668/) (2020)

## Built on by (most cited first, 27 of 27)

- [TextGrad: Automatic "Differentiation" via Text](https://mlsanity.com/paper/2406.07496/) (2024)
- [RouterBench: A Benchmark for Multi-LLM Routing System](https://mlsanity.com/paper/2403.12031/) (2024)
- [Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences](https://mlsanity.com/paper/2404.12272/) (2024)
- [A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agents](https://mlsanity.com/paper/2601.12538/) (2026)
- [Aviary: training language agents on challenging scientific tasks](https://mlsanity.com/paper/2412.21154/) (2024)
- [StructuredRAG: JSON Response Formatting with Large Language Models](https://mlsanity.com/paper/2408.11061/) (2024)
- [SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelines](https://mlsanity.com/paper/2401.03038/) (2024)
- [A Perspective for Adapting Generalist AI to Specialized Medical AI Applications and Their Challenges](https://mlsanity.com/paper/2411.00024/) (2024)
- [ChainBuddy: An AI Agent System for Generating LLM Pipelines](https://mlsanity.com/paper/2409.13588/) (2024)
- [ZeroSumEval: An Extensible Framework For Scaling LLM Evaluation with Inter-Model Competition](https://mlsanity.com/paper/2503.10673/) (2025)
- [ZeroSumEval: Scaling LLM Evaluation with Inter-Model Competition](https://mlsanity.com/paper/2504.12562/) (2025)
- [Making Prompts First-Class Citizens for Adaptive LLM Pipelines](https://mlsanity.com/paper/2508.05012/) (2025)
- [Automated QoR improvement in OpenROAD with coding agents](https://mlsanity.com/paper/2601.06268/) (2026)
- [Querying Databases with Function Calling](https://mlsanity.com/paper/2502.00032/) (2025)
- [Reconstructing Item Characteristic Curves using Fine-Tuned Large Language Models](https://mlsanity.com/paper/2601.02580/) (2026)
- [Probabilistic Programs of Thought](https://mlsanity.com/paper/2604.17290/) (2026)
- [Runtime-Structured Task Decomposition for Agentic Coding Systems](https://mlsanity.com/paper/2605.15425/) (2026)
- [Liberal Entity Matching as a Compound AI Toolchain](https://mlsanity.com/paper/2406.11255/) (2024)
- [Semantic Integrity Constraints: Declarative Guardrails for AI-Augmented Data Processing Systems](https://mlsanity.com/paper/2503.00600/) (2025)
- [FoodTaxo: Generating Food Taxonomies with Large Language Models](https://mlsanity.com/paper/2505.19838/) (2025)
- [Data Diversification Methods In Alignment Enhance Math Performance In LLMs](https://mlsanity.com/paper/2507.02173/) (2025)
- [Repairing Language Model Pipelines by Meta Self-Refining Competing Constraints at Runtime](https://mlsanity.com/paper/2507.10590/) (2025)
- [A Large-Scale Dataset and Citation Intent Classification in Turkish with LLMs](https://mlsanity.com/paper/2509.21907/) (2025)
- [What Helps Language Models Predict Human Beliefs: Demographics or Prior Stances?](https://mlsanity.com/paper/2511.18616/) (2025)
- [BiomechAgent: AI-Assisted Biomechanical Analysis Through Code-Generating Agents](https://mlsanity.com/paper/2602.06975/) (2026)
- [Harness as an Asset: Enforcing Determinism via the Convergent AI Agent Framework (CAAF)](https://mlsanity.com/paper/2604.17025/) (2026)
- [A Hybrid Dependency-Aware Framework for Task Decomposition and Dynamic Agent Generation in Oracle-to-PostgreSQL Migration](https://mlsanity.com/paper/2609.14413/) (2026)

## Abstract (arXiv)

Chaining language model (LM) calls as composable modules is fueling a new way of programming, but ensuring LMs adhere to important constraints requires heuristic "prompt engineering". We introduce LM Assertions, a programming construct for expressing computational constraints that LMs should satisfy. We integrate our constructs into the recent DSPy programming model for LMs, and present new strategies that allow DSPy to compile programs with LM Assertions into more reliable and accurate systems. We also propose strategies to use assertions at inference time for automatic self-refinement with LMs. We report on four diverse case studies for text generation and find that LM Assertions improve not only compliance with imposed rules but also downstream task performance, passing constraints up to 164% more often and generating up to 37% more higher-quality responses. Our reference implementation of LM Assertions is integrated into DSPy at https://github.com/stanfordnlp/dspy