mlsanityData to 7 Oct 2026

DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines

arXiv 2312.13382 ↗First posted 20 Dec 2023cs.CL cs.AI cs.PLcited by 27

Not on the map. Why?

Abstract (arXiv)

Chaining language model (LM) calls as composable modules is fueling a new way of programming, but ensuring LMs adhere to important constraints requires heuristic "prompt engineering". We introduce LM Assertions, a programming construct for expressing computational constraints that LMs should satisfy. We integrate our constructs into the recent DSPy programming model for LMs, and present new strategies that allow DSPy to compile programs with LM Assertions into more reliable and accurate systems. We also propose strategies to use assertions at inference time for automatic self-refinement with LMs. We report on four diverse case studies for text generation and find that LM Assertions improve not only compliance with imposed rules but also downstream task performance, passing constraints up to 164% more often and generating up to 37% more higher-quality responses. Our reference implementation of LM Assertions is integrated into DSPy at https://github.com/stanfordnlp/dspy

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 18 of its 38 references that are in the map. First posted on arXiv 20 Dec 2023.

Builds on

18 of 38 references.

References · 18

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  4. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  5. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  6. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
  7. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  8. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  9. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 316
  10. 2017Lexically Constrained Decoding for Sequence Generation Using Grid Beam Searchcited by 234
  11. 2022Prompting Is Programming: A Query Language for Large Language Modelscited by 110
  12. 2023Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinementcited by 89
6 more
  1. 2023LLMs cannot find reasoning errors, but can correct them given the error locationcited by 85
  2. 2023TrueTeacher: Learning Factual Consistency Evaluation with Large Language Modelscited by 82
  3. 2023Automatic Evaluation of Attribution by Large Language Modelscited by 70
  4. 2021Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrievalcited by 63
  5. 2023LLMRefine: Pinpointing and Refining Large Language Models via Fine-Grained Actionable Feedbackcited by 39
  6. 2020Model Assertions for Monitoring and Improving ML Modelscited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 10; 2026: 8.

27 papers.

Sort:

By area:

  1. 2024TextGrad: Automatic "Differentiation" via Textcited by 250
  2. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 222
  3. 2024Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferencescited by 142
  4. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  5. 2024Aviary: training language agents on challenging scientific taskscited by 23
  6. 2024StructuredRAG: JSON Response Formatting with Large Language Modelscited by 22
  7. 2024SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelinescited by 19
  8. 2024A Perspective for Adapting Generalist AI to Specialized Medical AI Applications and Their Challengescited by 8
  9. 2024ChainBuddy: An AI Agent System for Generating LLM Pipelinescited by 6
  10. 2025ZeroSumEval: An Extensible Framework For Scaling LLM Evaluation with Inter-Model Competitioncited by 5
  11. 2025ZeroSumEval: Scaling LLM Evaluation with Inter-Model Competitioncited by 5
  12. 2025Making Prompts First-Class Citizens for Adaptive LLM Pipelinescited by 3
15 more
  1. 2026Automated QoR improvement in OpenROAD with coding agentscited by 3
  2. 2025Querying Databases with Function Callingcited by 2
  3. 2026Reconstructing Item Characteristic Curves using Fine-Tuned Large Language Modelscited by 1
  4. 2026Probabilistic Programs of Thoughtcited by 1
  5. 2026Runtime-Structured Task Decomposition for Agentic Coding Systemscited by 1
  6. 2024Liberal Entity Matching as a Compound AI Toolchaincited by 0
  7. 2025Semantic Integrity Constraints: Declarative Guardrails for AI-Augmented Data Processing Systemscited by 0
  8. 2025FoodTaxo: Generating Food Taxonomies with Large Language Modelscited by 0
  9. 2025Data Diversification Methods In Alignment Enhance Math Performance In LLMscited by 0
  10. 2025Repairing Language Model Pipelines by Meta Self-Refining Competing Constraints at Runtimecited by 0
  11. 2025A Large-Scale Dataset and Citation Intent Classification in Turkish with LLMscited by 0
  12. 2025What Helps Language Models Predict Human Beliefs: Demographics or Prior Stances?cited by 0
  13. 2026BiomechAgent: AI-Assisted Biomechanical Analysis Through Code-Generating Agentscited by 0
  14. 2026Harness as an Asset: Enforcing Determinism via the Convergent AI Agent Framework (CAAF)cited by 0
  15. 2026A Hybrid Dependency-Aware Framework for Task Decomposition and Dynamic Agent Generation in Oracle-to-PostgreSQL Migrationcited by 0