# Revisiting Alternative Experimental Settings for Evaluating Top-N Item Recommendation Algorithms

arXiv: https://arxiv.org/abs/2010.04484 · HTML: https://mlsanity.com/paper/2010.04484/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Oct 2020.

## Built on by (most cited first, 24 of 24)

- [Towards Universal Sequence Representation Learning for Recommender Systems](https://mlsanity.com/paper/2206.05941/) (2022)
- [How to Index Item IDs for Recommendation Foundation Models](https://mlsanity.com/paper/2305.06569/) (2023)
- [Filter-enhanced MLP is All You Need for Sequential Recommendation](https://mlsanity.com/paper/2202.13556/) (2022)
- [Improving Graph Collaborative Filtering with Neighborhood-enriched Contrastive Learning](https://mlsanity.com/paper/2202.06200/) (2022)
- [User-oriented Fairness in Recommendation](https://mlsanity.com/paper/2104.10671/) (2021)
- [Personalized Counterfactual Fairness in Recommendation](https://mlsanity.com/paper/2105.09829/) (2021)
- [CORE: Simple and Effective Session-based Recommendation within Consistent Representation Space](https://mlsanity.com/paper/2204.11067/) (2022)
- [SLMRec: Distilling Large Language Models into Small for Sequential Recommendation](https://mlsanity.com/paper/2405.17890/) (2024)
- [Causal Collaborative Filtering](https://mlsanity.com/paper/2102.01868/) (2021)
- [On Manipulating Signals of User-Item Graph: A Jacobi Polynomial-based Graph Collaborative Filtering](https://mlsanity.com/paper/2306.03624/) (2023)
- [Popularity-Aware Alignment and Contrast for Mitigating Popularity Bias](https://mlsanity.com/paper/2405.20718/) (2024)
- [GUESR: A Global Unsupervised Data-Enhancement with Bucket-Cluster Sampling for Sequential Recommendation](https://mlsanity.com/paper/2303.00243/) (2023)
- [Towards Open-world Cross-Domain Sequential Recommendation: A Model-Agnostic Contrastive Denoising Approach](https://mlsanity.com/paper/2311.04760/) (2023)
- [Counterfactual Collaborative Reasoning](https://mlsanity.com/paper/2307.00165/) (2023)
- [Mixed Supervised Graph Contrastive Learning for Recommendation](https://mlsanity.com/paper/2404.15954/) (2024)
- [Do We Really Need Graph Convolution During Training? Light Post-Training Graph-ODE for Efficient Recommendation](https://mlsanity.com/paper/2407.18910/) (2024)
- [Dynamic Causal Collaborative Filtering](https://mlsanity.com/paper/2208.11094/) (2022)
- [Neuro-Symbolic Recommendation Model based on Logic Query](https://mlsanity.com/paper/2309.07594/) (2023)
- [Topology-aware Debiased Self-supervised Graph Learning for Recommendation](https://mlsanity.com/paper/2310.15858/) (2023)
- [RecSys Arena: Pair-wise Recommender System Evaluation with Large Language Models](https://mlsanity.com/paper/2412.11068/) (2024)
- [Recommender Transformers with Behavior Pathways](https://mlsanity.com/paper/2206.06804/) (2022)
- [On the Sweet Spot of Contrastive Views for Knowledge-enhanced Recommendation](https://mlsanity.com/paper/2309.13384/) (2023)
- [Dual Adversarial Perturbators Generate rich Views for Recommendation](https://mlsanity.com/paper/2409.06719/) (2024)
- [Lighter-X: An Efficient and Plug-and-play Strategy for Graph-based Recommendation through Decoupled Propagation](https://mlsanity.com/paper/2510.10105/) (2025)

## Abstract (arXiv)

Top-N item recommendation has been a widely studied task from implicit feedback. Although much progress has been made with neural methods, there is increasing concern on appropriate evaluation of recommendation algorithms. In this paper, we revisit alternative experimental settings for evaluating top-N recommendation algorithms, considering three important factors, namely dataset splitting, sampled metrics and domain selection. We select eight representative recommendation algorithms (covering both traditional and neural methods) and construct extensive experiments on a very large dataset. By carefully revisiting different options, we make several important findings on the three factors, which directly provide useful suggestions on how to appropriately set up the experiments for top-N item recommendation.