# Software Engineering for AI-Based Systems: A Survey

arXiv: https://arxiv.org/abs/2105.01984 · HTML: https://mlsanity.com/paper/2105.01984/

## Key facts

- Cited by 69 papers in the map (data through 14 Oct 2026).
- Builds on 83 of its 236 references that are in the map.
- First posted on arXiv 5 May 2021.

## Builds on (most cited first, 30 of 83)

- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Stealing Machine Learning Models via Prediction APIs](https://mlsanity.com/paper/1609.02943/) (2016)
- [Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing](https://mlsanity.com/paper/2001.00973/) (2020)
- [Safety Verification of Deep Neural Networks](https://mlsanity.com/paper/1610.06940/) (2016)
- [Improving fairness in machine learning systems: What do industry practitioners need?](https://mlsanity.com/paper/1812.05239/) (2018)
- [DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars](https://mlsanity.com/paper/1708.08559/) (2017)
- [Machine Learning Testing: Survey, Landscapes and Horizons](https://mlsanity.com/paper/1906.10742/) (2019)
- [Clipper: A Low-Latency Online Prediction Serving System](https://mlsanity.com/paper/1612.03079/) (2016)
- [Formal Security Analysis of Neural Networks using Symbolic Intervals](https://mlsanity.com/paper/1804.10829/) (2018)
- [Efficient Formal Safety Analysis of Neural Networks](https://mlsanity.com/paper/1809.08098/) (2018)
- [On a Formal Model of Safe and Scalable Self-driving Cars](https://mlsanity.com/paper/1708.06374/) (2017)
- [Trial without Error: Towards Safe Reinforcement Learning via Human Intervention](https://mlsanity.com/paper/1707.05173/) (2017)
- [Ethical Challenges in Data-Driven Dialogue Systems](https://mlsanity.com/paper/1711.09050/) (2017)
- [Automated Directed Fairness Testing](https://mlsanity.com/paper/1807.00468/) (2018)
- [Robust Physical-World Attacks on Deep Learning Models](https://mlsanity.com/paper/1707.08945/) (2017)
- [TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing](https://mlsanity.com/paper/1807.10875/) (2018)
- [A Comprehensive Study on Deep Learning Bug Characteristics](https://mlsanity.com/paper/1906.01388/) (2019)
- [On the Safety of Machine Learning: Cyber-Physical Systems, Decision Sciences, and Data Products](https://mlsanity.com/paper/1610.01256/) (2016)
- [Concolic Testing for Deep Neural Networks](https://mlsanity.com/paper/1805.00089/) (2018)
- [DeepBillboard: Systematic Physical-World Testing of Autonomous Driving Systems](https://mlsanity.com/paper/1812.10812/) (2018)
- [Towards Accountable AI: Hybrid Human-Machine Analyses for Characterizing System Failure](https://mlsanity.com/paper/1809.07424/) (2018)
- [Model-Reuse Attacks on Deep Learning Systems](https://mlsanity.com/paper/1812.00483/) (2018)
- [Requirements Engineering for Machine Learning: Perspectives from Data Scientists](https://mlsanity.com/paper/1908.04674/) (2019)
- [On Testing Machine Learning Programs](https://mlsanity.com/paper/1812.02257/) (2018)
- [Machine Teaching: A New Paradigm for Building Machine Learning Systems](https://mlsanity.com/paper/1707.06742/) (2017)
- [Software Engineering Challenges of Deep Learning](https://mlsanity.com/paper/1810.12034/) (2018)
- [Towards Corner Case Detection for Autonomous Driving](https://mlsanity.com/paper/1902.09184/) (2019)
- [Formal Scenario-Based Testing of Autonomous Vehicles: From Simulation to the Real World](https://mlsanity.com/paper/2003.07739/) (2020)
- [Safely Entering the Deep: A Review of Verification and Validation for Machine Learning and a Challenge Elicitation in the Automotive Industry](https://mlsanity.com/paper/1812.05389/) (2018)

## Built on by (most cited first, 30 of 69)

- [A Survey on LLM-based Code Generation for Low-Resource and Domain-Specific Programming Languages](https://mlsanity.com/paper/2410.03981/) (2024)
- [Collaboration Challenges in Building ML-Enabled Systems: Communication, Documentation, Engineering, and Process](https://mlsanity.com/paper/2110.10234/) (2021)
- [Exploring the Carbon Footprint of Hugging Face's ML Models: A Repository Mining Study](https://mlsanity.com/paper/2305.11164/) (2023)
- [A Meta-Summary of Challenges in Building Products with ML Components -- Collecting Experiences from 4758+ Practitioners](https://mlsanity.com/paper/2304.00078/) (2023)
- [Analyzing the Evolution and Maintenance of ML Models on Hugging Face](https://mlsanity.com/paper/2311.13380/) (2023)
- [METAL: Metamorphic Testing Framework for Analyzing Large-Language Model Qualities](https://mlsanity.com/paper/2312.06056/) (2023)
- [Bugs in Machine Learning-based Systems: A Faultload Benchmark](https://mlsanity.com/paper/2206.12311/) (2022)
- [The Pipeline for the Continuous Development of Artificial Intelligence Models -- Current State of Research and Practice](https://mlsanity.com/paper/2301.09001/) (2023)
- [Bug Characterization in Machine Learning-based Systems](https://mlsanity.com/paper/2307.14512/) (2023)
- [A Survey on Failure Analysis and Fault Injection in AI Systems](https://mlsanity.com/paper/2407.00125/) (2024)
- [Data Cleaning and Machine Learning: A Systematic Literature Review](https://mlsanity.com/paper/2310.01765/) (2023)
- [Machine Learning for Software Engineering: A Tertiary Study](https://mlsanity.com/paper/2211.09425/) (2022)
- [Which Design Decisions in AI-enabled Mobile Applications Contribute to Greener AI?](https://mlsanity.com/paper/2109.15284/) (2021)
- [Towards a Change Taxonomy for Machine Learning Systems](https://mlsanity.com/paper/2203.11365/) (2022)
- [Can ChatGPT advance software testing intelligence? An experience report on metamorphic testing](https://mlsanity.com/paper/2310.19204/) (2023)
- [A Large-Scale Study of Model Integration in ML-Enabled Software Systems](https://mlsanity.com/paper/2408.06226/) (2024)
- [Software Engineering and Foundation Models: Insights from Industry Blogs Using a Jury of Foundation Models](https://mlsanity.com/paper/2410.09012/) (2024)
- [Deep Learning Library Testing: Definition, Methods and Challenges](https://mlsanity.com/paper/2404.17871/) (2024)
- [How Mature is Requirements Engineering for AI-based Systems? A Systematic Mapping Study on Practices, Challenges, and Future Research Directions](https://mlsanity.com/paper/2409.07192/) (2024)
- [Data Smells: Categories, Causes and Consequences, and Detection of Suspicious Data in AI-based Systems](https://mlsanity.com/paper/2203.10384/) (2022)
- [A Compositional Approach to Creating Architecture Frameworks with an Application to Distributed AI Systems](https://mlsanity.com/paper/2212.13570/) (2022)
- [Design Patterns for AI-based Systems: A Multivocal Literature Review and Pattern Repository](https://mlsanity.com/paper/2303.13173/) (2023)
- [User-Like Bots for Cognitive Automation: A Survey](https://mlsanity.com/paper/2311.12154/) (2023)
- [A Catalog of Fairness-Aware Practices in Machine Learning Engineering](https://mlsanity.com/paper/2408.16683/) (2024)
- [Leveraging LLMs for User Stories in AI Systems: UStAI Dataset](https://mlsanity.com/paper/2504.00513/) (2025)
- [Differential testing for machine learning: an analysis for classification algorithms beyond deep learning](https://mlsanity.com/paper/2207.11976/) (2022)
- [Leveraging Artificial Intelligence on Binary Code Comprehension](https://mlsanity.com/paper/2210.05103/) (2022)
- [Towards green AI-based software systems: an architecture-centric approach (GAISSA)](https://mlsanity.com/paper/2307.09964/) (2023)
- [How to Sustainably Monitor ML-Enabled Systems? Accuracy and Energy Efficiency Tradeoffs in Concept Drift Detection](https://mlsanity.com/paper/2404.19452/) (2024)
- [Naming the Pain in Machine Learning-Enabled Systems Engineering](https://mlsanity.com/paper/2406.04359/) (2024)

## Abstract (arXiv)

AI-based systems are software systems with functionalities enabled by at least one AI component (e.g., for image- and speech-recognition, and autonomous driving). AI-based systems are becoming pervasive in society due to advances in AI. However, there is limited synthesized knowledge on Software Engineering (SE) approaches for building, operating, and maintaining AI-based systems. To collect and analyze state-of-the-art knowledge about SE for AI-based systems, we conducted a systematic mapping study. We considered 248 studies published between January 2010 and March 2020. SE for AI-based systems is an emerging research area, where more than 2/3 of the studies have been published since 2018. The most studied properties of AI-based systems are dependability and safety. We identified multiple SE approaches for AI-based systems, which we classified according to the SWEBOK areas. Studies related to software testing and software quality are very prevalent, while areas like software maintenance seem neglected. Data-related issues are the most recurrent challenges. Our results are valuable for: researchers, to quickly understand the state of the art and learn which topics need more research; practitioners, to learn about the approaches and challenges that SE entails for AI-based systems; and, educators, to bridge the gap among SE and AI in their curricula.