# Continuous Integration of Machine Learning Models with ease.ml/ci: Towards a Rigorous Yet Practical Treatment

arXiv: https://arxiv.org/abs/1903.00278 · HTML: https://mlsanity.com/paper/1903.00278/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Mar 2019.

## Built on by (most cited first, 9 of 9)

- [A Data Quality-Driven View of MLOps](https://mlsanity.com/paper/2102.07750/) (2021)
- [Towards Data-centric Graph Machine Learning: Review and Outlook](https://mlsanity.com/paper/2309.10979/) (2023)
- [The Pipeline for the Continuous Development of Artificial Intelligence Models -- Current State of Research and Practice](https://mlsanity.com/paper/2301.09001/) (2023)
- [On Continuous Integration / Continuous Delivery for Automated Deployment of Machine Learning Models using MLOps](https://mlsanity.com/paper/2202.03541/) (2022)
- [Design Patterns for AI-based Systems: A Multivocal Literature Review and Pattern Repository](https://mlsanity.com/paper/2303.13173/) (2023)
- [Machine Learning Application Development: Practitioners' Insights](https://mlsanity.com/paper/2112.15277/) (2021)
- [Challenges and Opportunities of Using Transformer-Based Multi-Task Learning in NLP Through ML Lifecycle: A Survey](https://mlsanity.com/paper/2308.08234/) (2023)
- [Enabling Reproducibility and Meta-learning Through a Lifelong Database of Experiments (LDE)](https://mlsanity.com/paper/2202.10979/) (2022)
- [Active-Learning-as-a-Service: An Automatic and Efficient MLOps System for Data-Centric AI](https://mlsanity.com/paper/2207.09109/) (2022)

## Abstract (arXiv)

Continuous integration is an indispensable step of modern software engineering practices to systematically manage the life cycles of system development. Developing a machine learning model is no difference - it is an engineering process with a life cycle, including design, implementation, tuning, testing, and deployment. However, most, if not all, existing continuous integration engines do not support machine learning as first-class citizens. In this paper, we present ease.ml/ci, to our best knowledge, the first continuous integration system for machine learning. The challenge of building ease.ml/ci is to provide rigorous guarantees, e.g., single accuracy point error tolerance with 0.999 reliability, with a practical amount of labeling effort, e.g., 2K labels per test. We design a domain specific language that allows users to specify integration conditions with reliability constraints, and develop simple novel optimizations that can lower the number of labels required by up to two orders of magnitude for test conditions popularly used in real production systems.