mlsanityData to 14 Oct 2026

Identifying Offline Metrics that Predict Online Impact: A Pragmatic Strategy for Real-World Recommender Systems

arXiv 2507.09566 ↗First posted 13 Jul 2025cs.IR cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

A critical challenge in recommender systems is to establish reliable relationships between offline and online metrics that predict real-world performance. Motivated by recent advances in Pareto front approximation, we introduce a pragmatic strategy for identifying offline metrics that align with online impact. A key advantage of this approach is its ability to simultaneously serve multiple test groups, each with distinct offline performance metrics, in an online experiment controlled by a single model. The method is model-agnostic for systems with a neural network backbone, enabling broad applicability across architectures and domains. We validate the strategy through a large-scale online experiment in the field of session-based recommender systems on the OTTO e-commerce platform. The online experiment identifies significant alignments between offline metrics and real-word click-through rate, post-click conversion rate and units sold. Our strategy provides industry practitioners with a valuable tool for understanding offline-to-online metric relationships and making informed, data-driven decisions.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 14 of its 28 references that are in the map. First posted on arXiv 13 Jul 2025.

Builds on

14 of 28 references.

References · 14

  1. 2015Session-based Recommendations with Recurrent Neural Networkscited by 695
  2. 2017Recurrent Neural Networks with Top-k Gains for Session-based Recommendationscited by 165
  3. 2016Off-policy evaluation for slate recommendationcited by 103
  4. 2020Counterfactual Evaluation of Slate Recommendations with Sequential Reward Interactionscited by 35
  5. 2019Measuring the Business Value of Recommender Systemscited by 32
  6. 2023Widespread Flaws in Offline Evaluation of Recommender Systemscited by 22
  7. 2023A Framework for Controllable Pareto Front Learning with Completed Scalarization Functions and its Applicationscited by 13
  8. 2020Debiased Off-Policy Evaluation for Recommendation Systemscited by 9
  9. 2023Scaling Session-Based Transformer Recommendations using Optimized Negative Sampling and Loss Functionscited by 9
  10. 2024Delta-rm OPE: Off-Policy Estimation with Pairs of Policiescited by 9
  11. 2018Off-line vs. On-line Evaluation of Recommender Systems in Small E-commercecited by 6
  12. 2024Pareto Front Approximation for Multi-Objective Session-Based Recommender Systemscited by 3
2 more
  1. 2022Offline Evaluation of Reward-Optimizing Recommender Systems: The Case of Simulationcited by 2
  2. 2023Bridging Offline-Online Evaluation with a Time-dependent and Popularity Bias-free Offline Metric for Recommenderscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026LLM-as-a-Judge for Reliable and Explainable Offline Evaluation in Top-K Recommendation ↗
  2. 2023On the Opportunities and Challenges of Offline Reinforcement Learning for Recommender Systems
  3. 2025Reward Balancing Revisited: Enhancing Offline Reinforcement Learning for Recommender Systems
  4. 2026From Prompting to Behavioral Alignment: Personalized LLM Judges for Recommendation Evaluation
  5. 2025A Universal Framework for Offline Serendipity Evaluation in Recommender Systems via Large Language Models ↗
  6. 2021On component interactions in two-stage recommender systems
  7. 2024A Comprehensive Review of Recommender Systems: Transitioning from Theory to Practice
  8. 2025Learning to Fast Unrank in Collaborative Filtering Recommendation