# Maintainability Challenges in ML: A Systematic Literature Review

arXiv: https://arxiv.org/abs/2408.09196 · HTML: https://mlsanity.com/paper/2408.09196/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 76 references that are in the map.
- First posted on arXiv 17 Aug 2024.

## Builds on (most cited first, 26 of 26)

- [Machine Learning Testing: Survey, Landscapes and Horizons](https://mlsanity.com/paper/1906.10742/) (2019)
- [Auto-Keras: An Efficient Neural Architecture Search System](https://mlsanity.com/paper/1806.10282/) (2018)
- [Challenges in Deploying Machine Learning: a Survey of Case Studies](https://mlsanity.com/paper/2011.09926/) (2020)
- [Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure](https://mlsanity.com/paper/2010.13561/) (2020)
- [Tunability: Importance of Hyperparameters of Machine Learning Algorithms](https://mlsanity.com/paper/1802.09596/) (2018)
- [On Testing Machine Learning Programs](https://mlsanity.com/paper/1812.02257/) (2018)
- [A Survey on Large-scale Machine Learning](https://mlsanity.com/paper/2008.03911/) (2020)
- [Adoption and Effects of Software Engineering Best Practices in Machine Learning](https://mlsanity.com/paper/2007.14130/) (2020)
- [AI Lifecycle Models Need To Be Revised. An Exploratory Study in Fintech](https://mlsanity.com/paper/2010.02716/) (2020)
- [Overton: A Data System for Monitoring and Improving Machine-Learned Products](https://mlsanity.com/paper/1909.05372/) (2019)
- [Characterizing Technical Debt and Antipatterns in AI-Based Systems: A Systematic Mapping Study](https://mlsanity.com/paper/2103.09783/) (2021)
- [An Empirical Study on Deployment Faults of Deep Learning Based Mobile Applications](https://mlsanity.com/paper/2101.04930/) (2021)
- [Quality Assurance for AI-based Systems: Overview and Challenges](https://mlsanity.com/paper/2102.05351/) (2021)
- [Software engineering for artificial intelligence and machine learning software: A systematic literature review](https://mlsanity.com/paper/2011.03751/) (2020)
- [500+ Times Faster Than Deep Learning (A Case Study Exploring Faster Methods for Text Mining StackOverflow)](https://mlsanity.com/paper/1802.05319/) (2018)
- [The Machine Learning Bazaar: Harnessing the ML Ecosystem for Effective System Development](https://mlsanity.com/paper/1905.08942/) (2019)
- [Software Architecture for ML-based Systems: What Exists and What Lies Ahead](https://mlsanity.com/paper/2103.07950/) (2021)
- [Data Cleaning for Accurate, Fair, and Robust Models: A Big Data - AI Integration Approach](https://mlsanity.com/paper/1904.10761/) (2019)
- [MLOps Challenges in Multi-Organization Setup: Experiences from Two Real-World Cases](https://mlsanity.com/paper/2103.08937/) (2021)
- [High Performance Data Engineering Everywhere](https://mlsanity.com/paper/2007.09589/) (2020)
- [Auptimizer -- an Extensible, Open-Source Framework for Hyperparameter Tuning](https://mlsanity.com/paper/1911.02522/) (2019)
- [Quality Management of Machine Learning Systems](https://mlsanity.com/paper/2006.09529/) (2020)
- [On the experiences of adopting automated data validation in an industrial machine learning project](https://mlsanity.com/paper/2103.04095/) (2021)
- [On Misbehaviour and Fault Tolerance in Machine Learning Systems](https://mlsanity.com/paper/2109.07989/) (2021)
- [HyperNOMAD: Hyperparameter optimization of deep neural networks using mesh adaptive direct search](https://mlsanity.com/paper/1907.01698/) (2019)
- [Shuffler: A Large Scale Data Management Tool for ML in Computer Vision](https://mlsanity.com/paper/2104.05125/) (2021)

## Built on by (most cited first, 5 of 5)

- [Machine Learning Operations: A Mapping Study](https://mlsanity.com/paper/2409.19416/) (2024)
- [Scalability and Maintainability Challenges and Solutions in Machine Learning: Systematic Literature Review](https://mlsanity.com/paper/2504.11079/) (2025)
- [HarmonE: A Self-Adaptive Approach to Architecting Sustainable MLOps](https://mlsanity.com/paper/2505.13693/) (2025)
- [Data-aware Static Analysis: Improving Detection of Semantic Faults in Machine Learning Code Using Data Characteristics](https://mlsanity.com/paper/2606.09957/) (2026)
- [From AI Technical Debt to Agentic Technical Debt: A Systematic Mapping of Root Causes and Manifestations in Agentic AI Systems](https://mlsanity.com/paper/2608.01001/) (2026)

## Abstract (arXiv)

Background: As Machine Learning (ML) advances rapidly in many fields, it is being adopted by academics and businesses alike. However, ML has a number of different challenges in terms of maintenance not found in traditional software projects. Identifying what causes these maintainability challenges can help mitigate them early and continue delivering value in the long run without degrading ML performance. Aim: This study aims to identify and synthesise the maintainability challenges in different stages of the ML workflow and understand how these stages are interdependent and impact each other's maintainability. Method: Using a systematic literature review, we screened more than 13000 papers, then selected and qualitatively analysed 56 of them. Results: (i) a catalogue of maintainability challenges in different stages of Data Engineering, Model Engineering workflows and the current challenges when building ML systems are discussed; (ii) a map of 13 maintainability challenges to different interdependent stages of ML that impact the overall workflow; (iii) Provided insights to developers of ML tools and researchers. Conclusions: In this study, practitioners and organisations will learn about maintainability challenges and their impact at different stages of ML workflow. This will enable them to avoid pitfalls and help to build a maintainable ML system. The implications and challenges will also serve as a basis for future research to strengthen our understanding of the ML system's maintainability.