# Are We Really Measuring Progress? Transferring Insights from Evaluating Recommender Systems to Temporal Link Prediction

arXiv: https://arxiv.org/abs/2506.12588 · HTML: https://mlsanity.com/paper/2506.12588/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 37 references that are in the map.
- First posted on arXiv 14 Jun 2025.

## Builds on (most cited first, 15 of 15)

- [Inductive Representation Learning on Temporal Graphs](https://mlsanity.com/paper/2002.07962/) (2020)
- [Predicting Dynamic Embedding Trajectory in Temporal Interaction Networks](https://mlsanity.com/paper/1908.01207/) (2019)
- [Inductive Representation Learning in Temporal Networks via Causal Anonymous Walks](https://mlsanity.com/paper/2101.05974/) (2021)
- [Towards Better Dynamic Graph Learning: New Architecture and Unified Library](https://mlsanity.com/paper/2303.13047/) (2023)
- [Temporal Graph Benchmark for Machine Learning on Temporal Graphs](https://mlsanity.com/paper/2307.01026/) (2023)
- [Optimized Cost per Click in Taobao Display Advertising](https://mlsanity.com/paper/1703.02091/) (2017)
- [Position: Graph Learning Will Lose Relevance Due To Poor Benchmarks](https://mlsanity.com/paper/2502.14546/) (2025)
- [Long Range Propagation on Continuous-Time Dynamic Graphs](https://mlsanity.com/paper/2406.02740/) (2024)
- [A Case Study on Sampling Strategies for Evaluating Neural Sequential Item Recommendation Models](https://mlsanity.com/paper/2107.13045/) (2021)
- [Widespread Flaws in Offline Evaluation of Recommender Systems](https://mlsanity.com/paper/2307.14951/) (2023)
- [Quality Metrics in Recommender Systems: Do We Calculate Metrics Consistently?](https://mlsanity.com/paper/2206.12858/) (2022)
- [TGB-Seq Benchmark: Challenging Temporal GNNs with Complex Sequential Dynamics](https://mlsanity.com/paper/2502.02975/) (2025)
- [On the Power of Heuristics in Temporal Graphs](https://mlsanity.com/paper/2502.04910/) (2025)
- [On Estimating Recommendation Evaluation Metrics under Sampling](https://mlsanity.com/paper/2103.01474/) (2021)
- [Towards Reliable Item Sampling for Recommendation Evaluation](https://mlsanity.com/paper/2211.15743/) (2022)

## Built on by (most cited first, 2 of 2)

- [What Do Temporal Graph Learning Models Learn?](https://mlsanity.com/paper/2510.09416/) (2025)
- [Do Temporal Link Predictors Need Learned Memory? A Smoothed-Count Baseline with a Handful of Parameters](https://mlsanity.com/paper/2609.35364/) (2026)

## Abstract (arXiv)

Recent work has questioned the reliability of graph learning benchmarks, citing concerns around task design, methodological rigor, and data suitability. In this extended abstract, we contribute to this discussion by focusing on evaluation strategies in Temporal Link Prediction (TLP). We observe that current evaluation protocols are often affected by one or more of the following issues: (1) inconsistent sampled metrics, (2) reliance on hard negative sampling often introduced as a means to improve robustness, and (3) metrics that implicitly assume equal base probabilities across source nodes by combining predictions. We support these claims through illustrative examples and connections to longstanding concerns in the recommender systems community. Our ongoing work aims to systematically characterize these problems and explore alternatives that can lead to more robust and interpretable evaluation. We conclude with a discussion of potential directions for improving the reliability of TLP benchmarks.