# The Simpson's Paradox in the Offline Evaluation of Recommendation Systems

arXiv: https://arxiv.org/abs/2104.08912 · HTML: https://mlsanity.com/paper/2104.08912/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Apr 2021.

## Built on by (most cited first, 16 of 16)

- [On (Normalised) Discounted Cumulative Gain as an Off-Policy Evaluation Metric for Top-n Recommendation](https://mlsanity.com/paper/2307.15053/) (2023)
- [Simpson's Paradox in Recommender Fairness: Reconciling differences between per-user and aggregated evaluations](https://mlsanity.com/paper/2210.07755/) (2022)
- [Offline Recommender System Evaluation under Unobserved Confounding](https://mlsanity.com/paper/2309.04222/) (2023)
- [Active Inference and Human--Computer Interaction](https://mlsanity.com/paper/2412.14741/) (2024)
- [Fashion-AlterEval: A Dataset for Improved Evaluation of Conversational Recommendation Systems with Alternative Relevant Items](https://mlsanity.com/paper/2507.18017/) (2025)
- [Recommender Systems: A Primer](https://mlsanity.com/paper/2302.02579/) (2023)
- [Separating and Learning Latent Confounders to Enhancing User Preferences Modeling](https://mlsanity.com/paper/2311.03381/) (2023)
- [Exploring Popularity Bias in Session-based Recommendation](https://mlsanity.com/paper/2312.07855/) (2023)
- [Counterfactual Evaluation of Ads Ranking Models through Domain Adaptation](https://mlsanity.com/paper/2409.19824/) (2024)
- [Confounding is a Pervasive Problem in Real World Recommender Systems](https://mlsanity.com/paper/2508.10479/) (2025)
- [Causality-aware Graph Aggregation Weight Estimator for Popularity Debiasing in Top-K Recommendation](https://mlsanity.com/paper/2510.04502/) (2025)
- [An Interpretable CF-RL-TOPSIS Fusion Model for Skills-Aware Talent Recommendation](https://mlsanity.com/paper/2605.24155/) (2026)
- [Distributional Approximate Nearest Neighbour Search for Uncertainty-Aware Retrieval](https://mlsanity.com/paper/2606.04603/) (2026)
- [NxN E-valuation: Hypothesis Certification via a Conformal CRT Null](https://mlsanity.com/paper/2608.06621/) (2026)
- [The Laws of Context Allocation: Causal Measurement and Closed-Loop Orchestration in Generative Search](https://mlsanity.com/paper/2608.23252/) (2026)
- [AURA: Agentic Diagnosis and Refinement for Production Recommender Systems at Scale](https://mlsanity.com/paper/2609.16625/) (2026)

## Abstract (arXiv)

Recommendation systems are often evaluated based on user's interactions that were collected from an existing, already deployed recommendation system. In this situation, users only provide feedback on the exposed items and they may not leave feedback on other items since they have not been exposed to them by the deployed system. As a result, the collected feedback dataset that is used to evaluate a new model is influenced by the deployed system, as a form of closed loop feedback. In this paper, we show that the typical offline evaluation of recommender systems suffers from the so-called Simpson's paradox. Simpson's paradox is the name given to a phenomenon observed when a significant trend appears in several different sub-populations of observational data but disappears or is even reversed when these sub-populations are combined together. Our in-depth experiments based on stratified sampling reveal that a very small minority of items that are frequently exposed by the deployed system plays a confounding factor in the offline evaluation of recommendation systems. In addition, we propose a novel evaluation methodology that takes into account the confounder, i.e the deployed system's characteristics. Using the relative comparison of many recommendation models as in the typical offline evaluation of recommender systems, and based on the Kendall rank correlation coefficient, we show that our proposed evaluation methodology exhibits statistically significant improvements of 14% and 40% on the examined open loop datasets (Yahoo! and Coat), respectively, in reflecting the true ranking of systems with an open loop (randomised) evaluation in comparison to the standard evaluation.