# KuaiRec: A Fully-observed Dataset and Insights for Evaluating Recommender Systems

arXiv: https://arxiv.org/abs/2202.10842 · HTML: https://mlsanity.com/paper/2202.10842/

## Key facts

- Cited by 96 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Feb 2022.

## Built on by (most cited first, 30 of 96)

- [Generate What You Prefer: Reshaping Sequential Recommendation via Guided Diffusion](https://mlsanity.com/paper/2310.20453/) (2023)
- [Binary Classification with Confidence Difference](https://mlsanity.com/paper/2310.05632/) (2023)
- [Generative Regression Based Watch Time Prediction for Short-Video Recommendation](https://mlsanity.com/paper/2412.20211/) (2024)
- [Scalable Neural Contextual Bandit for Recommender Systems](https://mlsanity.com/paper/2306.14834/) (2023)
- [From Variability to Stability: Advancing RecSys Benchmarking Practices](https://mlsanity.com/paper/2402.09766/) (2024)
- [Relevance meets Diversity: A User-Centric Framework for Knowledge Exploration through Recommendations](https://mlsanity.com/paper/2408.03772/) (2024)
- [HyperBandit: Contextual Bandit with Hypernewtork for Time-Varying User Preferences in Streaming Recommendation](https://mlsanity.com/paper/2308.08497/) (2023)
- [EasyRL4Rec: An Easy-to-use Library for Reinforcement Learning Based Recommender Systems](https://mlsanity.com/paper/2402.15164/) (2024)
- [Towards Real-world Human Behavior Simulation: Benchmarking Large Language Models on Long-horizon, Cross-scenario, Heterogeneous Behavior Traces](https://mlsanity.com/paper/2604.08362/) (2026)
- [Debiased Recommendation with Noisy Feedback](https://mlsanity.com/paper/2406.17182/) (2024)
- [Effective Off-Policy Evaluation and Learning in Contextual Combinatorial Bandits](https://mlsanity.com/paper/2408.11202/) (2024)
- [Doubly Calibrated Estimator for Recommendation on Data Missing Not At Random](https://mlsanity.com/paper/2403.00817/) (2024)
- [Policy Learning for Balancing Short-Term and Long-Term Rewards](https://mlsanity.com/paper/2405.03329/) (2024)
- [KuaiLive: A Real-time Interactive Dataset for Live Streaming Recommendation](https://mlsanity.com/paper/2508.05633/) (2025)
- [MUSE: A Simple Yet Effective Multimodal Search-Based Framework for Lifelong User Interest Modeling](https://mlsanity.com/paper/2512.07216/) (2025)
- [Bayesian Off-Policy Evaluation and Learning for Large Action Spaces](https://mlsanity.com/paper/2402.14664/) (2024)
- [Cache-Aware Reinforcement Learning in Large-Scale Recommender Systems](https://mlsanity.com/paper/2404.14961/) (2024)
- [ROLeR: Effective Reward Shaping in Offline Reinforcement Learning for Recommender Systems](https://mlsanity.com/paper/2407.13163/) (2024)
- [Qilin: A Multimodal Information Retrieval Dataset with APP-level User Sessions](https://mlsanity.com/paper/2503.00501/) (2025)
- [Addressing Missing Data Issue for Diffusion-based Recommendation](https://mlsanity.com/paper/2505.12283/) (2025)
- [Martingale Posterior Neural Networks for Fast Sequential Decision Making](https://mlsanity.com/paper/2506.11898/) (2025)
- [CTR-Sink: Attention Sink for Language Models in Click-Through Rate Prediction](https://mlsanity.com/paper/2508.03668/) (2025)
- [Mirroring Users: Towards Building Preference-aligned User Simulator with User Feedback in Recommendation](https://mlsanity.com/paper/2508.18142/) (2025)
- [PACEvolve++: Improving Test-time Learning for Evolutionary Search Agents](https://mlsanity.com/paper/2605.07039/) (2026)
- [Uncertainty-Aware Instance Reweighting for Off-Policy Learning](https://mlsanity.com/paper/2303.06389/) (2023)
- [Diffusion Models in Recommendation Systems: A Survey](https://mlsanity.com/paper/2501.10548/) (2025)
- [GFlowGR: Fine-tuning Generative Recommendation Frameworks with Generative Flow Networks](https://mlsanity.com/paper/2506.16114/) (2025)
- [On Efficiency-Effectiveness Trade-off of Diffusion-based Recommenders](https://mlsanity.com/paper/2510.17245/) (2025)
- [HiSAC: Hierarchical Sparse Activation Compression for Ultra-long Sequence Modeling in Recommenders](https://mlsanity.com/paper/2602.21009/) (2026)
- [GISTBench: Evaluating LLM User Understanding via Evidence-Based Interest Verification](https://mlsanity.com/paper/2603.29112/) (2026)

## Abstract (arXiv)

The progress of recommender systems is hampered mainly by evaluation as it requires real-time interactions between humans and systems, which is too laborious and expensive. This issue is usually approached by utilizing the interaction history to conduct offline evaluation. However, existing datasets of user-item interactions are partially observed, leaving it unclear how and to what extent the missing interactions will influence the evaluation. To answer this question, we collect a fully-observed dataset from Kuaishou's online environment, where almost all 1,411 users have been exposed to all 3,327 items. To the best of our knowledge, this is the first real-world fully-observed data with millions of user-item interactions. With this unique dataset, we conduct a preliminary analysis of how the two factors - data density and exposure bias - affect the evaluation results of multi-round conversational recommendation. Our main discoveries are that the performance ranking of different methods varies with the two factors, and this effect can only be alleviated in certain cases by estimating missing interactions for user simulation. This demonstrates the necessity of the fully-observed dataset. We release the dataset and the pipeline implementation for evaluation at https://kuairec.com