# Auto-Validate by-History: Auto-Program Data Quality Constraints to Validate Recurring Data Pipelines

arXiv: https://arxiv.org/abs/2306.02421 · HTML: https://mlsanity.com/paper/2306.02421/

## Key facts

- Cited by 5 papers in the map (data through 7 Oct 2026).
- Builds on 8 of its 75 references that are in the map.
- First posted on arXiv 4 Jun 2023.

## Builds on (most cited first, 8 of 8)

- [Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding](https://mlsanity.com/paper/1802.04431/) (2018)
- [LSTM-based Encoder-Decoder for Multi-sensor Anomaly Detection](https://mlsanity.com/paper/1607.00148/) (2016)
- [Time-Series Anomaly Detection Service at Microsoft](https://mlsanity.com/paper/1906.03821/) (2019)
- [ECOD: Unsupervised Outlier Detection Using Empirical Cumulative Distribution Functions](https://mlsanity.com/paper/2201.00382/) (2022)
- [HoloDetect: Few-Shot Learning for Error Detection](https://mlsanity.com/paper/1904.02285/) (2019)
- [Auto-FuzzyJoin: Auto-Program Fuzzy Similarity Joins Without Labeled Examples](https://mlsanity.com/paper/2103.04489/) (2021)
- [Auto-Validate: Unsupervised Data Validation Using Data-Domain Patterns Inferred from Data Lakes](https://mlsanity.com/paper/2104.04659/) (2021)
- [Effective and Complete Discovery of Order Dependencies via Set-based Axiomatization](https://mlsanity.com/paper/1608.06169/) (2016)

## Built on by (most cited first, 5 of 5)

- [Auto-Test: Learning Semantic-Domain Constraints for Unsupervised Error Detection in Tables](https://mlsanity.com/paper/2504.10762/) (2025)
- [Machine Learning Operations: A Mapping Study](https://mlsanity.com/paper/2409.19416/) (2024)
- [From Chaos to Clarity: Schema-Constrained AI for Auditable Biomedical Evidence Extraction from Full-Text PDFs](https://mlsanity.com/paper/2601.14267/) (2025)
- [Demonstrating Data-to-Knowledge Pipelines for Connecting Production Sites in the World Wide Lab](https://mlsanity.com/paper/2412.12231/) (2024)
- [Autonomous Data Processing using Meta-Agents](https://mlsanity.com/paper/2602.00307/) (2026)

## Abstract (arXiv)

Data pipelines are widely employed in modern enterprises to power a variety of Machine-Learning (ML) and Business-Intelligence (BI) applications. Crucially, these pipelines are recurring (e.g., daily or hourly) in production settings to keep data updated so that ML models can be re-trained regularly, and BI dashboards refreshed frequently. However, data quality (DQ) issues can often creep into recurring pipelines because of upstream schema and data drift over time. As modern enterprises operate thousands of recurring pipelines, today data engineers have to spend substantial efforts to manually monitor and resolve DQ issues, as part of their DataOps and MLOps practices. Given the high human cost of managing large-scale pipeline operations, it is imperative that we can automate as much as possible. In this work, we propose Auto-Validate-by-History (AVH) that can automatically detect DQ issues in recurring pipelines, leveraging rich statistics from historical executions. We formalize this as an optimization problem, and develop constant-factor approximation algorithms with provable precision guarantees. Extensive evaluations using 2000 production data pipelines at Microsoft demonstrate the effectiveness and efficiency of AVH.