# Towards a Change Taxonomy for Machine Learning Systems

arXiv: https://arxiv.org/abs/2203.11365 · HTML: https://mlsanity.com/paper/2203.11365/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 69 references that are in the map.
- First posted on arXiv 21 Mar 2022.

## Builds on (most cited first, 14 of 14)

- [Software Engineering for AI-Based Systems: A Survey](https://mlsanity.com/paper/2105.01984/) (2021)
- [Collaboration Challenges in Building ML-Enabled Systems: Communication, Documentation, Engineering, and Process](https://mlsanity.com/paper/2110.10234/) (2021)
- [Software Engineering Challenges of Deep Learning](https://mlsanity.com/paper/1810.12034/) (2018)
- [Identifying Implementation Bugs in Machine Learning based Image Classifiers using Metamorphic Testing](https://mlsanity.com/paper/1808.05353/) (2018)
- [What's in a GitHub Star? Understanding Repository Starring Practices in a Social Coding Platform](https://mlsanity.com/paper/1811.07643/) (2018)
- [Studying Software Engineering Patterns for Designing Machine Learning Systems](https://mlsanity.com/paper/1910.04736/) (2019)
- [An Empirical Comparison of Dependency Network Evolution in Seven Software Packaging Ecosystems](https://mlsanity.com/paper/1710.04936/) (2017)
- [Better Code, Better Sharing:On the Need of Analyzing Jupyter Notebooks](https://mlsanity.com/paper/1906.05234/) (2019)
- [Detecting and Understanding Real-World Differential Performance Bugs in Machine Learning Libraries](https://mlsanity.com/paper/2006.01991/) (2020)
- [Asset Management in Machine Learning: A Survey](https://mlsanity.com/paper/2102.06919/) (2021)
- [What Makes a Popular Academic AI Repository?](https://mlsanity.com/paper/2010.02472/) (2020)
- [Coding Together at Scale: GitHub as a Collaborative Social Network](https://mlsanity.com/paper/1407.2535/) (2014)
- [An Insight into the Pull Requests of GitHub](https://mlsanity.com/paper/1807.01853/) (2018)
- [Which Pull Requests Get Accepted and Why? A study of popular NPM Packages](https://mlsanity.com/paper/2003.01153/) (2020)

## Built on by (most cited first, 6 of 6)

- [SPICE: An Automated SWE-Bench Labeling Pipeline for Issue Clarity, Test Coverage, and Effort Estimation](https://mlsanity.com/paper/2507.09108/) (2025)
- [Challenges and Practices of Deep Learning Model Reengineering: A Case Study on Computer Vision](https://mlsanity.com/paper/2303.07476/) (2023)
- [PeaTMOSS: A Dataset and Initial Analysis of Pre-Trained Models in Open-Source Software](https://mlsanity.com/paper/2402.00699/) (2024)
- [How do Machine Learning Models Change?](https://mlsanity.com/paper/2411.09645/) (2024)
- [Model Context Protocols in Adaptive Transport Systems: A Survey](https://mlsanity.com/paper/2508.19239/) (2025)
- [From Questions to Insights: Exploring XAI Challenges Reported on Stack Overflow Questions](https://mlsanity.com/paper/2504.03085/) (2025)

## Abstract (arXiv)

Machine Learning (ML) research publications commonly provide open-source implementations on GitHub, allowing their audience to replicate, validate, or even extend machine learning algorithms, data sets, and metadata. However, thus far little is known about the degree of collaboration activity happening on such ML research repositories, in particular regarding (1) the degree to which such repositories receive contributions from forks, (2) the nature of such contributions (i.e., the types of changes), and (3) the nature of changes that are not contributed back to forks, which might represent missed opportunities. In this paper, we empirically study contributions to 1,346 ML research repositories and their 67,369 forks, both quantitatively and qualitatively (by building on Hindle et al.'s seminal taxonomy of code changes). We found that while ML research repositories are heavily forked, only 9% of the forks made modifications to the forked repository. 42% of the latter sent changes to the parent repositories, half of which (52%) were accepted by the parent repositories. Our qualitative analysis on 539 contributed and 378 local (fork-only) changes, extends Hindle et al.'s taxonomy with one new top-level change category related to ML (Data), and 15 new sub-categories, including nine ML-specific ones (input data, output data, program data, sharing, change evaluation, parameter tuning, performance, pre-processing, model training). While the changes that are not contributed back by the forks mostly concern domain-specific customizations and local experimentation (e.g., parameter tuning), the origin ML repositories do miss out on a non-negligible 15.4% of Documentation changes, 13.6% of Feature changes and 11.4% of Bug fix changes. The findings in this paper will be useful for practitioners, researchers, toolsmiths, and educators.