# The Effect of Third Party Implementations on Reproducibility

arXiv: https://arxiv.org/abs/2307.14956 · HTML: https://mlsanity.com/paper/2307.14956/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jul 2023.

## Built on by (most cited first, 10 of 10)

- [From Variability to Stability: Advancing RecSys Benchmarking Practices](https://mlsanity.com/paper/2402.09766/) (2024)
- [Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenders](https://mlsanity.com/paper/2507.16289/) (2025)
- [Recommender Systems for Good (RS4Good): Survey of Use Cases and a Call to Action for Research that Matters](https://mlsanity.com/paper/2411.16645/) (2024)
- [Performance Comparison of Session-based Recommendation Algorithms based on GNNs](https://mlsanity.com/paper/2312.16695/) (2023)
- [Investigating Reproducibility in Deep Learning-Based Software Fault Prediction](https://mlsanity.com/paper/2402.05645/) (2024)
- [Dataset-Agnostic Recommender Systems](https://mlsanity.com/paper/2501.07294/) (2025)
- [Reproducibility and Artifact Consistency of the SIGIR 2022 Recommender Systems Papers Based on Message Passing](https://mlsanity.com/paper/2503.07823/) (2025)
- [Universal Reusability in Recommender Systems: The Case for Dataset- and Task-Independent Frameworks](https://mlsanity.com/paper/2506.03391/) (2025)
- [Compositions of Variant Experts for Integrating Short-Term and Long-Term Preferences](https://mlsanity.com/paper/2506.23170/) (2025)
- [Reproducibility of Machine Learning-Based Fault Detection and Diagnosis for HVAC Systems in Buildings: An Empirical Study](https://mlsanity.com/paper/2508.00880/) (2025)

## Abstract (arXiv)

Reproducibility of recommender systems research has come under scrutiny during recent years. Along with works focusing on repeating experiments with certain algorithms, the research community has also started discussing various aspects of evaluation and how these affect reproducibility. We add a novel angle to this discussion by examining how unofficial third-party implementations could benefit or hinder reproducibility. Besides giving a general overview, we thoroughly examine six third-party implementations of a popular recommender algorithm and compare them to the official version on five public datasets. In the light of our alarming findings we aim to draw the attention of the research community to this neglected aspect of reproducibility.