# Characterizing and Detecting Mismatch in Machine-Learning-Enabled Systems

arXiv: https://arxiv.org/abs/2103.14101 · HTML: https://mlsanity.com/paper/2103.14101/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 21 references that are in the map.
- First posted on arXiv 25 Mar 2021.

## Builds on (most cited first, 8 of 8)

- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards](https://mlsanity.com/paper/1805.03677/) (2018)
- [Guidelines for including grey literature and conducting multivocal literature reviews in software engineering](https://mlsanity.com/paper/1707.02553/) (2017)
- [Adoption and Effects of Software Engineering Best Practices in Machine Learning](https://mlsanity.com/paper/2007.14130/) (2020)
- [FactSheets: Increasing Trust in AI Services through Supplier's Declarations of Conformity](https://mlsanity.com/paper/1808.07261/) (2018)
- [A Methodology for Creating AI FactSheets](https://mlsanity.com/paper/2006.13796/) (2020)
- [Beyond the technical challenges for deploying Machine Learning solutions in a software company](https://mlsanity.com/paper/1708.02363/) (2017)

## Built on by (most cited first, 19 of 19)

- [Collaboration Challenges in Building ML-Enabled Systems: Communication, Documentation, Engineering, and Process](https://mlsanity.com/paper/2110.10234/) (2021)
- [A Meta-Summary of Challenges in Building Products with ML Components -- Collecting Experiences from 4758+ Practitioners](https://mlsanity.com/paper/2304.00078/) (2023)
- [Data Smells: Categories, Causes and Consequences, and Detection of Suspicious Data in AI-based Systems](https://mlsanity.com/paper/2203.10384/) (2022)
- [What are the Machine Learning best practices reported by practitioners on Stack Exchange?](https://mlsanity.com/paper/2301.10516/) (2023)
- [MLOps Spanning Whole Machine Learning Life Cycle: A Survey](https://mlsanity.com/paper/2304.07296/) (2023)
- [A Preliminary Investigation of MLOps Practices in GitHub](https://mlsanity.com/paper/2209.11453/) (2022)
- [MLTEing Models: Negotiating, Evaluating, and Documenting Model and System Qualities](https://mlsanity.com/paper/2303.01998/) (2023)
- [Assessing the Quality of Computational Notebooks for a Frictionless Transition from Exploration to Production](https://mlsanity.com/paper/2205.11941/) (2022)
- [Enhancing Architecture Frameworks by Including Modern Stakeholders and their Views/Viewpoints](https://mlsanity.com/paper/2308.05239/) (2023)
- [The Rise and Fall(?) of Software Engineering](https://mlsanity.com/paper/2406.10141/) (2024)
- [How Do Model Export Formats Impact the Development of ML-Enabled Systems? A Case Study on Model Integration](https://mlsanity.com/paper/2502.00429/) (2025)
- [Measuring AI Systems Beyond Accuracy](https://mlsanity.com/paper/2204.04211/) (2022)
- [Towards machine learning guided by best practices](https://mlsanity.com/paper/2305.00233/) (2023)
- [An Exploratory Study on Automatic Identification of Assumptions in the Development of Deep Learning Frameworks](https://mlsanity.com/paper/2401.03653/) (2024)
- [Using Quality Attribute Scenarios for ML Model Test Case Generation](https://mlsanity.com/paper/2406.08575/) (2024)
- [Perspective of Software Engineering Researchers on Machine Learning Practices Regarding Research, Review, and Education](https://mlsanity.com/paper/2411.19304/) (2024)
- [Understanding Practitioners Perspectives on Monitoring Machine Learning Systems](https://mlsanity.com/paper/2509.25195/) (2025)
- [Exploring CoCo Challenges in ML Engineering Teams: Insights From the Semiconductor Industry](https://mlsanity.com/paper/2605.07389/) (2026)
- [Large Language Models for Automated Cross-Domain Machine Learning Task Type Identification: A Benchmark Dataset and Evaluation](https://mlsanity.com/paper/2609.35335/) (2026)

## Abstract (arXiv)

Increasing availability of machine learning (ML) frameworks and tools, as well as their promise to improve solutions to data-driven decision problems, has resulted in popularity of using ML techniques in software systems. However, end-to-end development of ML-enabled systems, as well as their seamless deployment and operations, remain a challenge. One reason is that development and deployment of ML-enabled systems involves three distinct workflows, perspectives, and roles, which include data science, software engineering, and operations. These three distinct perspectives, when misaligned due to incorrect assumptions, cause ML mismatches which can result in failed systems. We conducted an interview and survey study where we collected and validated common types of mismatches that occur in end-to-end development of ML-enabled systems. Our analysis shows that how each role prioritizes the importance of relevant mismatches varies, potentially contributing to these mismatched assumptions. In addition, the mismatch categories we identified can be specified as machine readable descriptors contributing to improved ML-enabled system development. In this paper, we report our findings and their implications for improving end-to-end ML-enabled system development.