mlsanityData to 14 Oct 2026

Reproducibility and Artifact Consistency of the SIGIR 2022 Recommender Systems Papers Based on Message Passing

arXiv 2503.07823 ↗First posted 10 Mar 2025cs.IR cs.DL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Graph-based techniques relying on neural networks and embeddings have gained attention as a way to develop Recommender Systems (RS) with several papers on the topic presented at SIGIR 2022 and 2023. Given the importance of ensuring that published research is methodologically sound and reproducible, in this paper we analyze 10 graph-based RS papers, most of which were published at SIGIR 2022, and assess their impact on subsequent work published in SIGIR 2023. Our analysis reveals several critical points that require attention: (i) the prevalence of bad practices, such as erroneous data splits or information leakage between training and testing data, which call into question the validity of the results; (ii) frequent inconsistencies between the provided artifacts (source code and data) and their descriptions in the paper, causing uncertainty about what is actually being evaluated; and (iii) the preference for new or complex baselines that are weaker compared to simpler ones, creating the impression of continuous improvement even when, particularly for the Amazon-Book dataset, the state-of-the-art has significantly worsened. Due to these issues, we are unable to confirm the claims made in most of the papers that we examined and attempted to reproduce.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 43 of its 84 references that are in the map. First posted on arXiv 10 Mar 2025.

Builds on

43 of 84 references.

References · 43

  1. 2020LightGCN: Simplifying and Powering Graph Convolution Network for Recommendationcited by 1,139
  2. 2019Neural Graph Collaborative Filteringcited by 661
  3. 2020Graph Neural Networks in Recommender Systems: A Surveycited by 494
  4. 2019KGAT: Knowledge Graph Attention Network for Recommendationcited by 343
  5. 2020Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)cited by 322
  6. 2018Variational Autoencoders for Collaborative Filteringcited by 310
  7. 2014Predictive Entropy Search for Efficient Global Optimization of Black-box Functionscited by 204
  8. 2019Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approachescited by 185
  9. 2021Are we really making much progress? Revisiting, benchmarking, and refining heterogeneous graph neural networkscited by 176
  10. 2021Are Graph Augmentations Necessary? Simple Graph Contrastive Learning for Recommendationcited by 173
  11. 2020Neural Collaborative Filtering vs. Matrix Factorization Revisitedcited by 123
  12. 2020Knowledge Graph Embedding for Link Prediction: A Comparative Analysiscited by 85
31 more
  1. 2020Computing Graph Neural Networks: A Survey from Algorithms to Acceleratorscited by 81
  2. 2022Knowledge Graph Contrastive Learning for Recommendationcited by 70
  3. 2019A Troubling Analysis of Reproducibility and Progress in Recommender Systems Researchcited by 65
  4. 2022Hypergraph Contrastive Collaborative Filteringcited by 63
  5. 2021A Survey on Embedding Dynamic Graphscited by 60
  6. 2021Graph Trend Filtering Networks for Recommendationscited by 53
  7. 2021How Powerful is Graph Convolution for Recommendation?cited by 50
  8. 2022Blurring-Sharpening Process Models for Collaborative Filteringcited by 35
  9. 2022Multi-level Cross-view Contrastive Learning for Knowledge-aware Recommender Systemcited by 30
  10. 2023Graph Transformer for Recommendationcited by 28
  11. 2021Simplified Data Wrangling with ir_datasetscited by 27
  12. 2019Critically Examining the "Neural Hype": Weak Baselines and the Additivity of Effectiveness Gains from Neural Ranking Modelscited by 24
  13. 2022Self-Guided Learning to Denoise for Robust Recommendationcited by 24
  14. 2021TREC Deep Learning Track: Reusable Test Collections in the Large Data Regimecited by 23
  15. 2023Disentangled Contrastive Collaborative Filteringcited by 22
  16. 2023Generative Contrastive Graph Learning for Recommendationcited by 20
  17. 2022HAKG: Hierarchy-Aware Knowledge Gated Network for Recommendationcited by 16
  18. 2021Significant Improvements over the State of the Art? A Case Study of the MS MARCO Document Ranking Leaderboardcited by 12
  19. 2023Triple Structural Information Modelling for Accurate, Explainable and Interactive Recommendationcited by 12
  20. 2022Less is More: Reweighting Important Spectral Graph Features for Recommendationcited by 10
  21. 2023The Effect of Third Party Implementations on Reproducibilitycited by 10
  22. 2023Knowledge-refined Denoising Network for Robust Recommendationcited by 9
  23. 2020Critically Examining the Claimed Value of Convolutions over User-Item Embedding Maps for Recommender Systemscited by 7
  24. 2020How to Measure the Reproducibility of System-oriented IR Experimentscited by 6
  25. 2021Reenvisioning Collaborative Filtering vs Matrix Factorizationcited by 6
  26. 2022Building a Culture of Reproducibility in Academic Researchcited by 5
  27. 2021INMO: A Model-Agnostic and Scalable Module for Inductive Collaborative Filteringcited by 4
  28. 2023Challenging the Myth of Graph Collaborative Filtering: a Reasoned and Reproducibility-driven Analysiscited by 3
  29. 2020Unify Local and Global Information for Top-N Recommendationcited by 2
  30. 2023Collaborative Residual Metric Learningcited by 2
  31. 2025A Worrying Reproducibility Study of Intent-Aware Recommendation Modelscited by 1

Built on by

Not cited yet.