# A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges

arXiv: https://arxiv.org/abs/2012.07919 · HTML: https://mlsanity.com/paper/2012.07919/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Dec 2020.

## Built on by (most cited first, 30 of 33)

- [Software Engineering for AI-Based Systems: A Survey](https://mlsanity.com/paper/2105.01984/) (2021)
- [A Survey on Machine Learning Techniques for Source Code Analysis](https://mlsanity.com/paper/2110.09610/) (2021)
- [A Meta-Summary of Challenges in Building Products with ML Components -- Collecting Experiences from 4758+ Practitioners](https://mlsanity.com/paper/2304.00078/) (2023)
- [Prompts Are Programs Too! Understanding How Developers Build Software Containing Prompts](https://mlsanity.com/paper/2409.12447/) (2024)
- [Agility in Software 2.0 -- Notebook Interfaces and MLOps with Buttresses and Rebars](https://mlsanity.com/paper/2111.14142/) (2021)
- [Test & Evaluation Best Practices for Machine Learning-Enabled Systems](https://mlsanity.com/paper/2310.06800/) (2023)
- [An Empirical Study of Challenges in Machine Learning Asset Management](https://mlsanity.com/paper/2402.15990/) (2024)
- [Differential testing for machine learning: an analysis for classification algorithms beyond deep learning](https://mlsanity.com/paper/2207.11976/) (2022)
- [Leveraging Artificial Intelligence on Binary Code Comprehension](https://mlsanity.com/paper/2210.05103/) (2022)
- [Machine Learning Systems: A Survey from a Data-Oriented Perspective](https://mlsanity.com/paper/2302.04810/) (2023)
- [Telecom AI Native Systems in the Age of Generative AI -- An Engineering Perspective](https://mlsanity.com/paper/2310.11770/) (2023)
- [Naming the Pain in Machine Learning-Enabled Systems Engineering](https://mlsanity.com/paper/2406.04359/) (2024)
- [Software Engineering Approaches for TinyML based IoT Embedded Vision: A Systematic Literature Review](https://mlsanity.com/paper/2204.08702/) (2022)
- [Modeling Quality and Machine Learning Pipelines through Extended Feature Models](https://mlsanity.com/paper/2207.07528/) (2022)
- [Bridging MDE and AI: A Systematic Review of Domain-Specific Languages and Model-Driven Practices in AI Software Systems Engineering](https://mlsanity.com/paper/2307.04599/) (2023)
- [LoCoML: A Framework for Real-World ML Inference Pipelines](https://mlsanity.com/paper/2501.14165/) (2025)
- [Explainable Artificial Intelligence Techniques for Software Development Lifecycle: A Phase-specific Survey](https://mlsanity.com/paper/2505.07058/) (2025)
- [A Metrics-Oriented Architectural Model to Characterize Complexity on Machine Learning-Enabled Systems](https://mlsanity.com/paper/2506.08153/) (2025)
- [The Semi-Executable Stack: Agentic Software Engineering and the Expanding Scope of SE](https://mlsanity.com/paper/2604.15468/) (2026)
- [Capturing Dependencies within Machine Learning via a Formal Process Model](https://mlsanity.com/paper/2208.05219/) (2022)
- [A Framework to Model ML Engineering Processes](https://mlsanity.com/paper/2404.18531/) (2024)
- [How Do Model Export Formats Impact the Development of ML-Enabled Systems? A Case Study on Model Integration](https://mlsanity.com/paper/2502.00429/) (2025)
- [Towards Requirements Engineering for RAG Systems](https://mlsanity.com/paper/2505.07553/) (2025)
- [Cataloguing Hugging Face Models to Software Engineering Activities: Automation and Findings](https://mlsanity.com/paper/2506.03013/) (2025)
- [Good Tools are Half the Work: Tool Usage in Deep Learning Projects](https://mlsanity.com/paper/2310.19124/) (2023)
- [On Security Weaknesses and Vulnerabilities in Deep Learning Systems](https://mlsanity.com/paper/2406.08688/) (2024)
- [How to Measure Human-AI Prediction Accuracy in Explainable AI Systems](https://mlsanity.com/paper/2409.00069/) (2024)
- [Maturity Framework for Enhancing Machine Learning Quality](https://mlsanity.com/paper/2502.15758/) (2025)
- [Scalability and Maintainability Challenges and Solutions in Machine Learning: Systematic Literature Review](https://mlsanity.com/paper/2504.11079/) (2025)
- [A Tale of Two Systems: Characterizing Architectural Complexity on Machine Learning-Enabled Systems](https://mlsanity.com/paper/2506.11295/) (2025)

## Abstract (arXiv)

Context: Advancements in machine learning (ML) lead to a shift from the traditional view of software development, where algorithms are hard-coded by humans, to ML systems materialized through learning from data. Therefore, we need to revisit our ways of developing software systems and consider the particularities required by these new types of systems. Objective: The purpose of this study is to systematically identify, analyze, summarize, and synthesize the current state of software engineering (SE) research for engineering ML systems. Method: I performed a systematic literature review (SLR). I systematically selected a pool of 141 studies from SE venues and then conducted a quantitative and qualitative analysis using the data extracted from these studies. Results: The non-deterministic nature of ML systems complicates all SE aspects of engineering ML systems. Despite increasing interest from 2018 onwards, the results reveal that none of the SE aspects have a mature set of tools and techniques. Testing is by far the most popular area among researchers. Even for testing ML systems, engineers have only some tool prototypes and solution proposals with weak experimental proof. Many of the challenges of ML systems engineering were identified through surveys and interviews. Researchers should conduct experiments and case studies, ideally in industrial environments, to further understand these challenges and propose solutions. Conclusion: The results may benefit (1) practitioners in foreseeing the challenges of ML systems engineering; (2) researchers and academicians in identifying potential research questions; and (3) educators in designing or updating SE courses to cover ML systems engineering.