# Engineering AI Systems: A Research Agenda

arXiv: https://arxiv.org/abs/2001.07522 · HTML: https://mlsanity.com/paper/2001.07522/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jan 2020.

## Built on by (most cited first, 30 of 37)

- [Software Engineering for AI-Based Systems: A Survey](https://mlsanity.com/paper/2105.01984/) (2021)
- [Collaboration Challenges in Building ML-Enabled Systems: Communication, Documentation, Engineering, and Process](https://mlsanity.com/paper/2110.10234/) (2021)
- [Characterizing Technical Debt and Antipatterns in AI-Based Systems: A Systematic Mapping Study](https://mlsanity.com/paper/2103.09783/) (2021)
- [Real-time End-to-End Federated Learning: An Automotive Case Study](https://mlsanity.com/paper/2103.11879/) (2021)
- [Ergo, SMIRK is Safe: A Safety Case for a Machine Learning Component in a Pedestrian Automatic Emergency Brake System](https://mlsanity.com/paper/2204.07874/) (2022)
- [Software Architecture for ML-based Systems: What Exists and What Lies Ahead](https://mlsanity.com/paper/2103.07950/) (2021)
- [Cross-Cloud Data Privacy Protection: Optimizing Collaborative Mechanisms of AI Systems by Integrating Federated Learning and LLMs](https://mlsanity.com/paper/2505.13292/) (2025)
- [Requirement Engineering Challenges for AI-intense Systems Development](https://mlsanity.com/paper/2103.10270/) (2021)
- [A Large-Scale Study of Model Integration in ML-Enabled Software Systems](https://mlsanity.com/paper/2408.06226/) (2024)
- [Data Smells in Public Datasets](https://mlsanity.com/paper/2203.08007/) (2022)
- [Artificial Intelligence in Industry 4.0: A Review of Integration Challenges for Industrial Systems](https://mlsanity.com/paper/2405.18580/) (2024)
- [Agility in Software 2.0 -- Notebook Interfaces and MLOps with Buttresses and Rebars](https://mlsanity.com/paper/2111.14142/) (2021)
- [Data Smells: Categories, Causes and Consequences, and Detection of Suspicious Data in AI-based Systems](https://mlsanity.com/paper/2203.10384/) (2022)
- [A Compositional Approach to Creating Architecture Frameworks with an Application to Distributed AI Systems](https://mlsanity.com/paper/2212.13570/) (2022)
- [Design Patterns for AI-based Systems: A Multivocal Literature Review and Pattern Repository](https://mlsanity.com/paper/2303.13173/) (2023)
- [User-Like Bots for Cognitive Automation: A Survey](https://mlsanity.com/paper/2311.12154/) (2023)
- [Lessons Learned from Educating AI Engineers](https://mlsanity.com/paper/2103.10703/) (2021)
- [Exploring ML testing in practice -- Lessons learned from an interactive rapid review with Axis Communications](https://mlsanity.com/paper/2203.16225/) (2022)
- [Engineering LLM Powered Multi-agent Framework for Autonomous CloudOps](https://mlsanity.com/paper/2501.08243/) (2025)
- [Towards a Reference Software Architecture for Human-AI Teaming in Smart Manufacturing](https://mlsanity.com/paper/2201.04876/) (2022)
- [Modeling Quality and Machine Learning Pipelines through Extended Feature Models](https://mlsanity.com/paper/2207.07528/) (2022)
- [What About the Data? A Mapping Study on Data Engineering for AI Systems](https://mlsanity.com/paper/2402.05156/) (2024)
- [LoCoML: A Framework for Real-World ML Inference Pipelines](https://mlsanity.com/paper/2501.14165/) (2025)
- [AI Total: Analyzing Security ML Models with Imperfect Data in Production](https://mlsanity.com/paper/2110.07028/) (2021)
- [Capturing Dependencies within Machine Learning via a Formal Process Model](https://mlsanity.com/paper/2208.05219/) (2022)
- [Defining Quality Requirements for a Trustworthy AI Wildflower Monitoring Platform](https://mlsanity.com/paper/2303.13151/) (2023)
- [A Framework to Model ML Engineering Processes](https://mlsanity.com/paper/2404.18531/) (2024)
- [How Do Model Export Formats Impact the Development of ML-Enabled Systems? A Case Study on Model Integration](https://mlsanity.com/paper/2502.00429/) (2025)
- [Agentic Problem Frames: A Systematic Approach to Engineering Reliable Domain Agents](https://mlsanity.com/paper/2602.19065/) (2026)
- [Engineering an Intelligent Essay Scoring and Feedback System: An Experience Report](https://mlsanity.com/paper/2103.13590/) (2021)

## Abstract (arXiv)

Artificial intelligence (AI) and machine learning (ML) are increasingly broadly adopted in industry, However, based on well over a dozen case studies, we have learned that deploying industry-strength, production quality ML models in systems proves to be challenging. Companies experience challenges related to data quality, design methods and processes, performance of models as well as deployment and compliance. We learned that a new, structured engineering approach is required to construct and evolve systems that contain ML/DL components. In this paper, we provide a conceptualization of the typical evolution patterns that companies experience when employing ML as well as an overview of the key problems experienced by the companies that we have studied. The main contribution of the paper is a research agenda for AI engineering that provides an overview of the key engineering challenges surrounding ML solutions and an overview of open items that need to be addressed by the research community at large.