mlsanityData to 7 Oct 2026

SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelines

arXiv 2401.03038 ↗First posted 5 Jan 2024cs.DB cs.SEcited by 19

On the map: Synthetic data and self-improvement for LLMs → Code and instruction data synthesis

Abstract (arXiv)

Large language models (LLMs) are being increasingly deployed as part of pipelines that repeatedly process or generate data of some sort. However, a common barrier to deployment are the frequent and often unpredictable errors that plague LLMs. Acknowledging the inevitability of these errors, we propose em data quality assertions to identify when LLMs may be making mistakes. We present SPADE, a method for automatically synthesizing data quality assertions that identify bad LLM outputs. We make the observation that developers often identify data quality issues during prototyping prior to deployment, and attempt to address them by adding instructions to the LLM prompt over time. SPADE therefore analyzes histories of prompt versions over time to create candidate assertion functions and then selects a minimal set that fulfills both coverage and accuracy requirements. In testing across nine different real-world LLM pipelines, SPADE efficiently reduces the number of assertions by 14% and decreases false failures by 21% when compared to simpler baselines. SPADE has been deployed as an offering within LangSmith, LangChain's LLM pipeline hub, and has been used to generate data quality assertions for over 2000 pipelines across a spectrum of industries.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Code and instruction data synthesis, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 29 of its 59 references that are in the map. First posted on arXiv 5 Jan 2024.

Builds on

29 of 59 references.

References · 29

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  5. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  6. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  7. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  8. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 347
  9. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  10. 2021AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Promptscited by 265
  11. 2023How is ChatGPT's behavior changing over time?cited by 265
  12. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 259
17 more
  1. 2022Prompting GPT-3 To Be Reliablecited by 258
  2. 2022Can Foundation Models Wrangle Your Data?cited by 173
  3. 2023Software Testing with Large Language Models: Survey, Landscape, and Visioncited by 163
  4. 2022Ask Me Anything: A simple strategy for prompting language modelscited by 155
  5. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  6. 2020Challenges in Deploying Machine Learning: a Survey of Case Studiescited by 151
  7. 2022PromptChainer: Chaining Large Language Model Prompts through Visual Programmingcited by 134
  8. 2023Wider and Deeper LLM Networks are Fairer LLM Evaluatorscited by 91
  9. 2021Is GPT-3 Text Indistinguishable from Human Text? Scarecrow: A Framework for Scrutinizing Machine Textcited by 89
  10. 2023EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteriacited by 73
  11. 2023DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelinescited by 27
  12. 2022Operationalizing Machine Learning: An Interview Studycited by 25
  13. 2023Prompt Sapper: A LLM-Empowered Production Tool for Building AI Chainscited by 24
  14. 2020Model Assertions for Monitoring and Improving ML Modelscited by 22
  15. 2023Designing LLM Chains by Adapting Techniques from Crowdsourcing Workflowscited by 17
  16. 2023Revisiting Prompt Engineering via Declarative Crowdsourcingcited by 12
  17. 2022Finding Label and Model Errors in Perception Data With Learned Observation Assertionscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 9; 2026: 3.

19 papers.

Sort:

By area:

  1. 2024DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processingcited by 49
  2. 2024Inverse Constitutional AI: Compressing Preferences into Principlescited by 39
  3. 2024What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Usecited by 24
  4. 2024StructuredRAG: JSON Response Formatting with Large Language Modelscited by 22
  5. 2024Beyond the Comfort Zone: Emerging Solutions to Overcome Challenges in Integrating LLMs into Software Productscited by 11
  6. 2025PromptPex: Automatic Test Generation for Language Model Promptscited by 9
  7. 2024ChainBuddy: An AI Agent System for Generating LLM Pipelinescited by 6
  8. 2026Test-Driven AI Agent Definition (TDAD): Compiling Tool-Using Agents from Behavioral Specificationscited by 6
  9. 2024LLM+KG@VLDB'24 Workshop Summarycited by 3
  10. 2025Making Prompts First-Class Citizens for Adaptive LLM Pipelinescited by 3
  11. 2025Querying Databases with Function Callingcited by 2
  12. 2025Adaptive Testing for LLM-Based Applications: A Diversity-based Approachcited by 1
7 more
  1. 2025Gensors: Authoring Personalized Visual Sensors with Multimodal Foundation Models and Reasoningcited by 1
  2. 2025Rating Quality of Diverse Time Series Data by Meta-learning from LLM Judgmentcited by 1
  3. 2025Lost in the Pipeline: How Well Do Large Language Models Handle Data Preparation?cited by 1
  4. 2026VISTA: Knowledge-Driven Vessel Trajectory Imputation with Repair Provenancecited by 1
  5. 2025Semantic Integrity Constraints: Declarative Guardrails for AI-Augmented Data Processing Systemscited by 0
  6. 2025Controlling Context: Generative AI at Work in Integrated Circuit Design and Other High-Precision Domainscited by 0
  7. 2026Stage-Audit: Auditable Source-Frontier Discovery for Cross-Wiki Tablescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Quality Matters: Evaluating Synthetic Data for Tool-Using LLMs
  2. 2026A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data
  3. 2026DataEvolver: Automatic Data Preparation for Large Language Models through Multi-Level Self-Evolving
  4. 2025DataSciBench: An LLM Agent Benchmark for Data Science
  5. 2026Know When To Fold 'Em: Token-Efficient LLM Synthetic Data Generation via Multi-Stage In-Flight Rejection
  6. 2025CLEAR: Error Analysis via LLM-as-a-Judge Made Easy
  7. 2025LoopTool: Closing the Data-Training Loop for Robust LLM Tool Calls
  8. 2024DataDreamer: A Tool for Synthetic Data Generation and Reproducible LLM Workflows