# An Experience Report on Machine Learning Reproducibility: Guidance for Practitioners and TensorFlow Model Garden Contributors

arXiv: https://arxiv.org/abs/2107.00821 · HTML: https://mlsanity.com/paper/2107.00821/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 42 references that are in the map.
- First posted on arXiv 2 Jul 2021.

## Builds on (most cited first, 10 of 10)

- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [CenterNet: Keypoint Triplets for Object Detection](https://mlsanity.com/paper/1904.08189/) (2019)
- [Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)](https://mlsanity.com/paper/2003.12206/) (2020)
- [Adoption and Effects of Software Engineering Best Practices in Machine Learning](https://mlsanity.com/paper/2007.14130/) (2020)
- [Studying Software Engineering Patterns for Designing Machine Learning Systems](https://mlsanity.com/paper/1910.04736/) (2019)
- [Machine Learning Software Engineering in Practice: An Industrial Case Study](https://mlsanity.com/paper/1906.07154/) (2019)
- [Software Engineering Practice in the Development of Deep Learning Applications](https://mlsanity.com/paper/1910.03156/) (2019)

## Built on by (most cited first, 6 of 6)

- [A Survey on Machine Learning Techniques for Source Code Analysis](https://mlsanity.com/paper/2110.09610/) (2021)
- [Challenges and Practices of Deep Learning Model Reengineering: A Case Study on Computer Vision](https://mlsanity.com/paper/2303.07476/) (2023)
- [Reproducibility of Machine Learning: Terminology, Recommendations and Open Issues](https://mlsanity.com/paper/2302.12691/) (2023)
- [Discrepancies among Pre-trained Deep Neural Networks: A New Threat to Model Zoo Reliability](https://mlsanity.com/paper/2303.02551/) (2023)
- [A Partial Replication of MaskFormer in TensorFlow on TPUs for the TensorFlow Model Garden](https://mlsanity.com/paper/2404.18801/) (2024)
- [Recommending Pre-Trained Models for IoT Devices](https://mlsanity.com/paper/2412.18972/) (2024)

## Abstract (arXiv)

Machine learning techniques are becoming a fundamental tool for scientific and engineering progress. These techniques are applied in contexts as diverse as astronomy and spam filtering. However, correctly applying these techniques requires careful engineering. Much attention has been paid to the technical potential; relatively little attention has been paid to the software engineering process required to bring research-based machine learning techniques into practical utility. Technology companies have supported the engineering community through machine learning frameworks such as TensorFLow and PyTorch, but the details of how to engineer complex machine learning models in these frameworks have remained hidden. To promote best practices within the engineering community, academic institutions and Google have partnered to launch a Special Interest Group on Machine Learning Models (SIGMODELS) whose goal is to develop exemplary implementations of prominent machine learning models in community locations such as the TensorFlow Model Garden (TFMG). The purpose of this report is to define a process for reproducing a state-of-the-art machine learning model at a level of quality suitable for inclusion in the TFMG. We define the engineering process and elaborate on each step, from paper analysis to model release. We report on our experiences implementing the YOLO model family with a team of 26 student researchers, share the tools we developed, and describe the lessons we learned along the way.