# Scalability and Maintainability Challenges and Solutions in Machine Learning: Systematic Literature Review

arXiv: https://arxiv.org/abs/2504.11079 · HTML: https://mlsanity.com/paper/2504.11079/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 46 of its 152 references that are in the map.
- First posted on arXiv 15 Apr 2025.

## Builds on (most cited first, 30 of 46)

- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://mlsanity.com/paper/1912.02781/) (2019)
- [DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale](https://mlsanity.com/paper/2201.05596/) (2022)
- [Kubric: A scalable dataset generator](https://mlsanity.com/paper/2203.03570/) (2022)
- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [Machine Learning Testing: Survey, Landscapes and Horizons](https://mlsanity.com/paper/1906.10742/) (2019)
- [A Survey on Distributed Machine Learning](https://mlsanity.com/paper/1912.09789/) (2019)
- [Auto-Keras: An Efficient Neural Architecture Search System](https://mlsanity.com/paper/1806.10282/) (2018)
- [Challenges in Deploying Machine Learning: a Survey of Case Studies](https://mlsanity.com/paper/2011.09926/) (2020)
- [Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure](https://mlsanity.com/paper/2010.13561/) (2020)
- [Tunability: Importance of Hyperparameters of Machine Learning Algorithms](https://mlsanity.com/paper/1802.09596/) (2018)
- [Collaboration Challenges in Building ML-Enabled Systems: Communication, Documentation, Engineering, and Process](https://mlsanity.com/paper/2110.10234/) (2021)
- [On Testing Machine Learning Programs](https://mlsanity.com/paper/1812.02257/) (2018)
- [Software Engineering Challenges of Deep Learning](https://mlsanity.com/paper/1810.12034/) (2018)
- [Towards CRISP-ML(Q): A Machine Learning Process Model with Quality Assurance Methodology](https://mlsanity.com/paper/2003.05155/) (2020)
- [Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Tools](https://mlsanity.com/paper/1903.11314/) (2019)
- [An Empirical Study towards Characterizing Deep Learning Development and Deployment across Different Frameworks and Platforms](https://mlsanity.com/paper/1909.06727/) (2019)
- [Adoption and Effects of Software Engineering Best Practices in Machine Learning](https://mlsanity.com/paper/2007.14130/) (2020)
- [AI Lifecycle Models Need To Be Revised. An Exploratory Study in Fintech](https://mlsanity.com/paper/2010.02716/) (2020)
- [A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges](https://mlsanity.com/paper/2012.07919/) (2020)
- [Overton: A Data System for Monitoring and Improving Machine-Learned Products](https://mlsanity.com/paper/1909.05372/) (2019)
- [An Empirical Study on Deployment Faults of Deep Learning Based Mobile Applications](https://mlsanity.com/paper/2101.04930/) (2021)
- [Quality Assurance for AI-based Systems: Overview and Challenges](https://mlsanity.com/paper/2102.05351/) (2021)
- [Scalable federated machine learning with FEDn](https://mlsanity.com/paper/2103.00148/) (2021)
- [MLCask: Efficient Management of Component Evolution in Collaborative Data Analytics Pipelines](https://mlsanity.com/paper/2010.10246/) (2020)
- [Software engineering for artificial intelligence and machine learning software: A systematic literature review](https://mlsanity.com/paper/2011.03751/) (2020)
- [500+ Times Faster Than Deep Learning (A Case Study Exploring Faster Methods for Text Mining StackOverflow)](https://mlsanity.com/paper/1802.05319/) (2018)
- [The Machine Learning Bazaar: Harnessing the ML Ecosystem for Effective System Development](https://mlsanity.com/paper/1905.08942/) (2019)
- [Data Cleaning for Accurate, Fair, and Robust Models: A Big Data - AI Integration Approach](https://mlsanity.com/paper/1904.10761/) (2019)

## Abstract (arXiv)

This systematic literature review examines the critical challenges and solutions related to scalability and maintainability in Machine Learning (ML) systems. As ML applications become increasingly complex and widespread across industries, the need to balance system scalability with long-term maintainability has emerged as a significant concern. This review synthesizes current research and practices addressing these dual challenges across the entire ML life-cycle, from data engineering to model deployment in production. We analyzed 124 papers to identify and categorize 41 maintainability challenges and 13 scalability challenges, along with their corresponding solutions. Our findings reveal intricate inter dependencies between scalability and maintainability, where improvements in one often impact the other. The review is structured around six primary research questions, examining maintainability and scalability challenges in data engineering, model engineering, and ML system development. We explore how these challenges manifest differently across various stages of the ML life-cycle. This comprehensive overview offers valuable insights for both researchers and practitioners in the field of ML systems. It aims to guide future research directions, inform best practices, and contribute to the development of more robust, efficient, and sustainable ML applications across various domains.