mlsanityData to 7 Oct 2026

The Pipeline for the Continuous Development of Artificial Intelligence Models -- Current State of Research and Practice

arXiv 2301.09001 ↗First posted 21 Jan 2023cs.SE cs.AIcited by 14

Not on the map. Why?

Abstract (arXiv)

Companies struggle to continuously develop and deploy AI models to complex production systems due to AI characteristics while assuring quality. To ease the development process, continuous pipelines for AI have become an active research area where consolidated and in-depth analysis regarding the terminology, triggers, tasks, and challenges is required. This paper includes a Multivocal Literature Review where we consolidated 151 relevant formal and informal sources. In addition, nine-semi structured interviews with participants from academia and industry verified and extended the obtained information. Based on these sources, this paper provides and compares terminologies for DevOps and CI/CD for AI, MLOps, (end-to-end) lifecycle management, and CD4ML. Furthermore, the paper provides an aggregated list of potential triggers for reiterating the pipeline, such as alert systems or schedules. In addition, this work uses a taxonomy creation strategy to present a consolidated pipeline comprising tasks regarding the continuous development of AI. This pipeline consists of four stages: Data Handling, Model Learning, Software Development and System Operations. Moreover, we map challenges regarding pipeline implementation, adaption, and usage for the continuous development of AI to these four stages.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 38 of its 177 references that are in the map. First posted on arXiv 21 Jan 2023.

Builds on

38 of 177 references.

References · 38

  1. 2020Challenges in Deploying Machine Learning: a Survey of Case Studiescited by 151
  2. 2022Machine Learning Operations (MLOps): Overview, Definition, and Architecturecited by 126
  3. 2017TensorFlow-Serving: Flexible, High-Performance ML Servingcited by 85
  4. 2021Software Engineering for AI-Based Systems: A Surveycited by 68
  5. 2017Guidelines for including grey literature and conducting multivocal literature reviews in software engineeringcited by 38
  6. 2021Who Needs MLOps: What Data Scientists Seek to Accomplish and How Can MLOps Help?cited by 37
  7. 2020AI Lifecycle Models Need To Be Revised. An Exploratory Study in Fintechcited by 33
  8. 2021A Data Quality-Driven View of MLOpscited by 31
  9. 2020Technology Readiness Levels for AI & MLcited by 30
  10. 2019Studying Software Engineering Patterns for Designing Machine Learning Systemscited by 24
  11. 2021Production Machine Learning Pipelines: Empirical Analysis and Optimization Opportunitiescited by 21
  12. 2021Quality Assurance for AI-based Systems: Overview and Challengescited by 19
26 more
  1. 2018DLHub: Model and Data Serving for Sciencecited by 17
  2. 2020MLModelCI: An Automatic Cloud Platform for Efficient MLaaScited by 16
  3. 2020Software engineering for artificial intelligence and machine learning software: A systematic literature reviewcited by 15
  4. 2020A Systematic Literature Review on Federated Machine Learning: From A Software Engineering Perspectivecited by 14
  5. 2016ModelHub: Towards Unified Data and Lifecycle Management for Deep Learningcited by 13
  6. 2020Developing and Operating Artificial Intelligence Models in Trustworthy Autonomous Systemscited by 10
  7. 2021MLOps Challenges in Multi-Organization Setup: Experiences from Two Real-World Casescited by 10
  8. 2019Continuous Integration of Machine Learning Models with ease.ml/ci: Towards a Rigorous Yet Practical Treatmentcited by 9
  9. 2019Engineering Reliable Deep Learning Systemscited by 6
  10. 2021Integration of Convolutional Neural Networks in Mobile Applicationscited by 6
  11. 2020The AIQ Meta-Testbed: Pragmatically Bridging Academic AI Testing and Industrial Q Needscited by 5
  12. 2021Reliable Fleet Analytics for Edge IoT Solutionscited by 4
  13. 2021Machine Learning Model Development from a Software Engineering Perspective: A Systematic Literature Reviewcited by 4
  14. 2019Towards Testing of Deep Learning Systems with Training Set Reductioncited by 3
  15. 2020Towards ML Engineering: A Brief History Of TensorFlow Extended (TFX)cited by 3
  16. 2019ISTHMUS: Secure, Scalable, Real-time and Robust Machine Learning Platform for Healthcarecited by 2
  17. 2020CodeReef: an open platform for portable MLOps, reusable automation actions and reproducible benchmarkingcited by 2
  18. 2020Towards Automating the AI Operations Lifecyclecited by 2
  19. 2020Green Lighting ML: Confidentiality, Integrity, and Availability of Machine Learning Systems in Deploymentcited by 2
  20. 2020ACCORDANT: A Domain Specific Model and DevOpsApproach for Big Data Analytics Architecturescited by 2
  21. 2021Systematic Mapping Study on the Machine Learning Lifecyclecited by 2
  22. 2019Quantitative Overfitting Management for Human-in-the-loop ML Application Development with ease.ml/metercited by 1
  23. 2020PipeSim: Trace-driven Simulation of Large-Scale AI Operations Platformscited by 1
  24. 2021Extending SOUP to ML Models When DesigningCertified Medical Systemscited by 1
  25. 2021Hidden Technical Debts for Fair Machine Learning in Financial Servicescited by 1
  26. 2021Concept for a Technical Infrastructure for Management of Predictive Models in Industrial Applicationscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 5; 2025: 3; 2026: 4.

14 papers.

Sort:

By area:

  1. 2023Modyn: Data-Centric Machine Learning Pipeline Orchestrationcited by 5
  2. 2025RAGOps: Operating and Managing Retrieval-Augmented Generation Pipelinescited by 5
  3. 2024Towards Trustworthy Machine Learning in Production: An Overview of the Robustness in MLOps Approachcited by 3
  4. 2025LoCoML: A Framework for Real-World ML Inference Pipelinescited by 2
  5. 2024A Survey of Pipeline Tools for Data Engineeringcited by 1
  6. 2024ALPACA -- Adaptive Learning Pipeline for Comprehensive AIcited by 1
  7. 2026How Do Practitioners Build SE Agents? Insights from a Mixed-Methods Studycited by 1
  8. 2023Quality Issues in Machine Learning Software Systemscited by 0
  9. 2024Standardizing Knowledge Engineering Practices with a Reference Architecturecited by 0
  10. 2024Being Accountable is Smart: Navigating the Technical and Regulatory Landscape of AI-based Services for Power Gridcited by 0
  11. 2025SLEAN: Simple Lightweight Ensemble Analysis Network for Multi-Provider LLM Coordination: Design, Implementation, and Vibe Coding Bug Investigation Case Studycited by 0
  12. 2026A Security-Oriented Lifecycle Model for Large Language Model Systemscited by 0
2 more
  1. 2026Software Engineering for AI-driven Building Operationcited by 0
  2. 2026SAREF-based Ontology for Distributed AI Workflows across the Edge-Fog-Cloud Continuumcited by 0