# The Pipeline for the Continuous Development of Artificial Intelligence Models -- Current State of Research and Practice

arXiv: https://arxiv.org/abs/2301.09001 · HTML: https://mlsanity.com/paper/2301.09001/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 177 references that are in the map.
- First posted on arXiv 21 Jan 2023.

## Builds on (most cited first, 30 of 38)

- [Challenges in Deploying Machine Learning: a Survey of Case Studies](https://mlsanity.com/paper/2011.09926/) (2020)
- [Machine Learning Operations (MLOps): Overview, Definition, and Architecture](https://mlsanity.com/paper/2205.02302/) (2022)
- [TensorFlow-Serving: Flexible, High-Performance ML Serving](https://mlsanity.com/paper/1712.06139/) (2017)
- [Software Engineering for AI-Based Systems: A Survey](https://mlsanity.com/paper/2105.01984/) (2021)
- [Guidelines for including grey literature and conducting multivocal literature reviews in software engineering](https://mlsanity.com/paper/1707.02553/) (2017)
- [Who Needs MLOps: What Data Scientists Seek to Accomplish and How Can MLOps Help?](https://mlsanity.com/paper/2103.08942/) (2021)
- [AI Lifecycle Models Need To Be Revised. An Exploratory Study in Fintech](https://mlsanity.com/paper/2010.02716/) (2020)
- [A Data Quality-Driven View of MLOps](https://mlsanity.com/paper/2102.07750/) (2021)
- [Technology Readiness Levels for AI & ML](https://mlsanity.com/paper/2006.12497/) (2020)
- [Studying Software Engineering Patterns for Designing Machine Learning Systems](https://mlsanity.com/paper/1910.04736/) (2019)
- [Production Machine Learning Pipelines: Empirical Analysis and Optimization Opportunities](https://mlsanity.com/paper/2103.16007/) (2021)
- [Quality Assurance for AI-based Systems: Overview and Challenges](https://mlsanity.com/paper/2102.05351/) (2021)
- [DLHub: Model and Data Serving for Science](https://mlsanity.com/paper/1811.11213/) (2018)
- [MLModelCI: An Automatic Cloud Platform for Efficient MLaaS](https://mlsanity.com/paper/2006.05096/) (2020)
- [Software engineering for artificial intelligence and machine learning software: A systematic literature review](https://mlsanity.com/paper/2011.03751/) (2020)
- [A Systematic Literature Review on Federated Machine Learning: From A Software Engineering Perspective](https://mlsanity.com/paper/2007.11354/) (2020)
- [ModelHub: Towards Unified Data and Lifecycle Management for Deep Learning](https://mlsanity.com/paper/1611.06224/) (2016)
- [Developing and Operating Artificial Intelligence Models in Trustworthy Autonomous Systems](https://mlsanity.com/paper/2003.05434/) (2020)
- [MLOps Challenges in Multi-Organization Setup: Experiences from Two Real-World Cases](https://mlsanity.com/paper/2103.08937/) (2021)
- [Continuous Integration of Machine Learning Models with ease.ml/ci: Towards a Rigorous Yet Practical Treatment](https://mlsanity.com/paper/1903.00278/) (2019)
- [Engineering Reliable Deep Learning Systems](https://mlsanity.com/paper/1910.12582/) (2019)
- [Integration of Convolutional Neural Networks in Mobile Applications](https://mlsanity.com/paper/2103.07286/) (2021)
- [The AIQ Meta-Testbed: Pragmatically Bridging Academic AI Testing and Industrial Q Needs](https://mlsanity.com/paper/2009.05260/) (2020)
- [Reliable Fleet Analytics for Edge IoT Solutions](https://mlsanity.com/paper/2101.04414/) (2021)
- [Machine Learning Model Development from a Software Engineering Perspective: A Systematic Literature Review](https://mlsanity.com/paper/2102.07574/) (2021)
- [Towards Testing of Deep Learning Systems with Training Set Reduction](https://mlsanity.com/paper/1901.04169/) (2019)
- [Towards ML Engineering: A Brief History Of TensorFlow Extended (TFX)](https://mlsanity.com/paper/2010.02013/) (2020)
- [ISTHMUS: Secure, Scalable, Real-time and Robust Machine Learning Platform for Healthcare](https://mlsanity.com/paper/1909.13343/) (2019)
- [CodeReef: an open platform for portable MLOps, reusable automation actions and reproducible benchmarking](https://mlsanity.com/paper/2001.07935/) (2020)
- [Towards Automating the AI Operations Lifecycle](https://mlsanity.com/paper/2003.12808/) (2020)

## Built on by (most cited first, 14 of 14)

- [Modyn: Data-Centric Machine Learning Pipeline Orchestration](https://mlsanity.com/paper/2312.06254/) (2023)
- [RAGOps: Operating and Managing Retrieval-Augmented Generation Pipelines](https://mlsanity.com/paper/2506.03401/) (2025)
- [Towards Trustworthy Machine Learning in Production: An Overview of the Robustness in MLOps Approach](https://mlsanity.com/paper/2410.21346/) (2024)
- [LoCoML: A Framework for Real-World ML Inference Pipelines](https://mlsanity.com/paper/2501.14165/) (2025)
- [How Do Practitioners Build SE Agents? Insights from a Mixed-Methods Study](https://mlsanity.com/paper/2607.10856/) (2026)
- [A Survey of Pipeline Tools for Data Engineering](https://mlsanity.com/paper/2406.08335/) (2024)
- [ALPACA -- Adaptive Learning Pipeline for Comprehensive AI](https://mlsanity.com/paper/2412.10950/) (2024)
- [Quality Issues in Machine Learning Software Systems](https://mlsanity.com/paper/2306.15007/) (2023)
- [Standardizing Knowledge Engineering Practices with a Reference Architecture](https://mlsanity.com/paper/2404.03624/) (2024)
- [Being Accountable is Smart: Navigating the Technical and Regulatory Landscape of AI-based Services for Power Grid](https://mlsanity.com/paper/2408.01121/) (2024)
- [SLEAN: Simple Lightweight Ensemble Analysis Network for Multi-Provider LLM Coordination: Design, Implementation, and Vibe Coding Bug Investigation Case Study](https://mlsanity.com/paper/2510.10010/) (2025)
- [A Security-Oriented Lifecycle Model for Large Language Model Systems](https://mlsanity.com/paper/2608.03626/) (2026)
- [Software Engineering for AI-driven Building Operation](https://mlsanity.com/paper/2608.16237/) (2026)
- [SAREF-based Ontology for Distributed AI Workflows across the Edge-Fog-Cloud Continuum](https://mlsanity.com/paper/2608.26160/) (2026)

## Abstract (arXiv)

Companies struggle to continuously develop and deploy AI models to complex production systems due to AI characteristics while assuring quality. To ease the development process, continuous pipelines for AI have become an active research area where consolidated and in-depth analysis regarding the terminology, triggers, tasks, and challenges is required. This paper includes a Multivocal Literature Review where we consolidated 151 relevant formal and informal sources. In addition, nine-semi structured interviews with participants from academia and industry verified and extended the obtained information. Based on these sources, this paper provides and compares terminologies for DevOps and CI/CD for AI, MLOps, (end-to-end) lifecycle management, and CD4ML. Furthermore, the paper provides an aggregated list of potential triggers for reiterating the pipeline, such as alert systems or schedules. In addition, this work uses a taxonomy creation strategy to present a consolidated pipeline comprising tasks regarding the continuous development of AI. This pipeline consists of four stages: Data Handling, Model Learning, Software Development and System Operations. Moreover, we map challenges regarding pipeline implementation, adaption, and usage for the continuous development of AI to these four stages.