# Identifying Offline Metrics that Predict Online Impact: A Pragmatic Strategy for Real-World Recommender Systems

arXiv: https://arxiv.org/abs/2507.09566 · HTML: https://mlsanity.com/paper/2507.09566/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 28 references that are in the map.
- First posted on arXiv 13 Jul 2025.

## Builds on (most cited first, 14 of 14)

- [Session-based Recommendations with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06939/) (2015)
- [Recurrent Neural Networks with Top-k Gains for Session-based Recommendations](https://mlsanity.com/paper/1706.03847/) (2017)
- [Off-policy evaluation for slate recommendation](https://mlsanity.com/paper/1605.04812/) (2016)
- [Counterfactual Evaluation of Slate Recommendations with Sequential Reward Interactions](https://mlsanity.com/paper/2007.12986/) (2020)
- [Measuring the Business Value of Recommender Systems](https://mlsanity.com/paper/1908.08328/) (2019)
- [Widespread Flaws in Offline Evaluation of Recommender Systems](https://mlsanity.com/paper/2307.14951/) (2023)
- [A Framework for Controllable Pareto Front Learning with Completed Scalarization Functions and its Applications](https://mlsanity.com/paper/2302.12487/) (2023)
- [Debiased Off-Policy Evaluation for Recommendation Systems](https://mlsanity.com/paper/2002.08536/) (2020)
- [Scaling Session-Based Transformer Recommendations using Optimized Negative Sampling and Loss Functions](https://mlsanity.com/paper/2307.14906/) (2023)
- [Delta-rm OPE: Off-Policy Estimation with Pairs of Policies](https://mlsanity.com/paper/2405.10024/) (2024)
- [Off-line vs. On-line Evaluation of Recommender Systems in Small E-commerce](https://mlsanity.com/paper/1809.03186/) (2018)
- [Pareto Front Approximation for Multi-Objective Session-Based Recommender Systems](https://mlsanity.com/paper/2407.16828/) (2024)
- [Offline Evaluation of Reward-Optimizing Recommender Systems: The Case of Simulation](https://mlsanity.com/paper/2209.08642/) (2022)
- [Bridging Offline-Online Evaluation with a Time-dependent and Popularity Bias-free Offline Metric for Recommenders](https://mlsanity.com/paper/2308.06885/) (2023)

## Abstract (arXiv)

A critical challenge in recommender systems is to establish reliable relationships between offline and online metrics that predict real-world performance. Motivated by recent advances in Pareto front approximation, we introduce a pragmatic strategy for identifying offline metrics that align with online impact. A key advantage of this approach is its ability to simultaneously serve multiple test groups, each with distinct offline performance metrics, in an online experiment controlled by a single model. The method is model-agnostic for systems with a neural network backbone, enabling broad applicability across architectures and domains. We validate the strategy through a large-scale online experiment in the field of session-based recommender systems on the OTTO e-commerce platform. The online experiment identifies significant alignments between offline metrics and real-word click-through rate, post-click conversion rate and units sold. Our strategy provides industry practitioners with a valuable tool for understanding offline-to-online metric relationships and making informed, data-driven decisions.