# CVTT: Cross-Validation Through Time

arXiv: https://arxiv.org/abs/2205.05393 · HTML: https://mlsanity.com/paper/2205.05393/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 74 references that are in the map.
- First posted on arXiv 11 May 2022.

## Builds on (most cited first, 26 of 26)

- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [Neural Collaborative Filtering](https://mlsanity.com/paper/1708.05031/) (2017)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [Neural Graph Collaborative Filtering](https://mlsanity.com/paper/1905.08108/) (2019)
- [KGAT: Knowledge Graph Attention Network for Recommendation](https://mlsanity.com/paper/1905.07854/) (2019)
- [Variational Autoencoders for Collaborative Filtering](https://mlsanity.com/paper/1802.05814/) (2018)
- [Reinforcement learning based recommender systems: A survey](https://mlsanity.com/paper/2101.06286/) (2021)
- [Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches](https://mlsanity.com/paper/1907.06902/) (2019)
- [Top-K Off-Policy Correction for a REINFORCE Recommender System](https://mlsanity.com/paper/1812.02353/) (2018)
- [VBPR: Visual Bayesian Personalized Ranking from Implicit Feedback](https://mlsanity.com/paper/1510.01784/) (2015)
- [Embarrassingly Shallow Autoencoders for Sparse Data](https://mlsanity.com/paper/1905.03375/) (2019)
- [Sequence-Aware Recommender Systems](https://mlsanity.com/paper/1802.08452/) (2018)
- [RepeatNet: A Repeat Aware Neural Recommendation Machine for Session-based Recommendation](https://mlsanity.com/paper/1812.02646/) (2018)
- [Exploring Data Splitting Strategies for the Evaluation of Recommendation Models](https://mlsanity.com/paper/2007.13237/) (2020)
- [How to Retrain Recommender System? A Sequential Meta-Learning Method](https://mlsanity.com/paper/2005.13258/) (2020)
- [A Critical Study on Data Leakage in Recommender System Offline Evaluation](https://mlsanity.com/paper/2010.11060/) (2020)
- [Sequential Variational Autoencoders for Collaborative Filtering](https://mlsanity.com/paper/1811.09975/) (2018)
- [Modeling Personalized Item Frequency Information for Next-basket Recommendation](https://mlsanity.com/paper/2006.00556/) (2020)
- [A Re-visit of the Popularity Baseline in Recommender Systems](https://mlsanity.com/paper/2005.13829/) (2020)
- [Fantastic Embeddings and How to Align Them: Zero-Shot Inference in a Multi-Shop Scenario](https://mlsanity.com/paper/2007.14906/) (2020)
- [Gated Attentive-Autoencoder for Content-Aware Recommendation](https://mlsanity.com/paper/1812.02869/) (2018)
- [Towards Amortized Ranking-Critical Training for Collaborative Filtering](https://mlsanity.com/paper/1906.04281/) (2019)
- [Exploring Lottery Ticket Hypothesis in Media Recommender Systems](https://mlsanity.com/paper/2108.00944/) (2021)
- [It's Time to Consider "Time" when Evaluating Recommender-System Algorithms [Proposal]](https://mlsanity.com/paper/1708.08447/) (2017)
- [Triple2Vec: Learning Triple Embeddings from Knowledge Graphs](https://mlsanity.com/paper/1905.11691/) (2019)
- [Variational Bayesian Context-aware Representation for Grocery Recommendation](https://mlsanity.com/paper/1909.07705/) (2019)

## Abstract (arXiv)

The evaluation of recommender systems from a practical perspective is a topic of ongoing discourse within the research community. While many current evaluation methods reduce performance to a single value metric as an easy way to compare models, it relies on the assumption that the methods' performance remains constant over time. In this study, we examine this assumption and propose the Cross-Validation Thought Time (CVTT) technique as a more comprehensive evaluation method, focusing on model performance over time. By utilizing the proposed technique, we conduct an in-depth analysis of the performance of popular RecSys algorithms. Our findings indicate that (1) the performance of the recommenders varies over time for all reviewed datasets, (2) using simple evaluation approaches can lead to a substantial decrease in performance in real-world evaluation scenarios, and (3) excessive data usage can lead to suboptimal results.