# Measuring the Business Value of Recommender Systems

arXiv: https://arxiv.org/abs/1908.08328 · HTML: https://mlsanity.com/paper/1908.08328/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Aug 2019.

## Built on by (most cited first, 30 of 32)

- [SimUSER: Simulating User Behavior with Large Language Models for Recommender System Evaluation](https://mlsanity.com/paper/2504.12722/) (2025)
- [Fairness in Recommender Systems: Research Landscape and Future Directions](https://mlsanity.com/paper/2205.11127/) (2022)
- [EB-NeRD: A Large-Scale Dataset for News Recommendation](https://mlsanity.com/paper/2410.03432/) (2024)
- [Generalized Delayed Feedback Model with Post-Click Information in Recommender Systems](https://mlsanity.com/paper/2206.00407/) (2022)
- [Doubly Calibrated Estimator for Recommendation on Data Missing Not At Random](https://mlsanity.com/paper/2403.00817/) (2024)
- [Balancing Consumer and Business Value of Recommender Systems: A Simulation-based Analysis](https://mlsanity.com/paper/2203.05952/) (2022)
- [AlignUSER: Human-Aligned LLM Agents via World Models for Recommender System Evaluation](https://mlsanity.com/paper/2601.00930/) (2026)
- [Synthetic Data-Based Simulators for Recommender Systems: A Survey](https://mlsanity.com/paper/2206.11338/) (2022)
- [Semi-supervised Adversarial Learning for Complementary Item Recommendation](https://mlsanity.com/paper/2303.05812/) (2023)
- [Recommender Systems for Sustainability: Overview and Research Issues](https://mlsanity.com/paper/2412.03620/) (2024)
- [Seven challenges for harmonizing explainability requirements](https://mlsanity.com/paper/2108.05390/) (2021)
- [Understanding Longitudinal Dynamics of Recommender Systems with Agent-Based Modeling and Simulation](https://mlsanity.com/paper/2108.11068/) (2021)
- [Sequential Recommendation Model for Next Purchase Prediction](https://mlsanity.com/paper/2207.06225/) (2022)
- [Calibrated Recommendations with Contextual Bandits](https://mlsanity.com/paper/2509.05460/) (2025)
- [Beyond Offline A/B Testing: Context-Aware Agent Simulation for Recommender System Evaluation](https://mlsanity.com/paper/2604.09549/) (2026)
- [FINEST: Stabilizing Recommendations by Rank-Preserving Fine-Tuning](https://mlsanity.com/paper/2402.03481/) (2024)
- [Recommender Systems for Good (RS4Good): Survey of Use Cases and a Call to Action for Research that Matters](https://mlsanity.com/paper/2411.16645/) (2024)
- [With a Little Help From My Friends: Collective Manipulation in Risk-Controlling Recommender Systems](https://mlsanity.com/paper/2603.28476/) (2026)
- [Online Evaluation Methods for the Causal Effect of Recommendations](https://mlsanity.com/paper/2107.06630/) (2021)
- [Multi-Objective Recommender Systems: Survey and Challenges](https://mlsanity.com/paper/2210.10309/) (2022)
- [Adversarial Text Rewriting for Text-aware Recommender Systems](https://mlsanity.com/paper/2408.00312/) (2024)
- [IntentRec: Predicting User Session Intent with Hierarchical Multi-Task Learning](https://mlsanity.com/paper/2408.05353/) (2024)
- [M-LLM3REC: A Motivation-Aware User-Item Interaction Framework for Enhancing Recommendation Accuracy with LLMs](https://mlsanity.com/paper/2508.15262/) (2025)
- [Evaluation of Agents under Simulated AI Marketplace Dynamics](https://mlsanity.com/paper/2604.14256/) (2026)
- [Recommender Systems: A Primer](https://mlsanity.com/paper/2302.02579/) (2023)
- [From Data to Decisions: The Transformational Power of Machine Learning in Business Recommendations](https://mlsanity.com/paper/2402.08109/) (2024)
- [EventChat: Implementation and user-centric evaluation of a large language model-driven conversational recommender system for exploring leisure events in an SME context](https://mlsanity.com/paper/2407.04472/) (2024)
- [Leveraging User-Generated Reviews for Recommender Systems with Dynamic Headers](https://mlsanity.com/paper/2409.07627/) (2024)
- [Identifying Offline Metrics that Predict Online Impact: A Pragmatic Strategy for Real-World Recommender Systems](https://mlsanity.com/paper/2507.09566/) (2025)
- [Controlled Personalization in Legacy Media Online Services: A Case Study in News Recommendation](https://mlsanity.com/paper/2510.09136/) (2025)

## Abstract (arXiv)

Recommender Systems are nowadays successfully used by all major web sites (from e-commerce to social media) to filter content and make suggestions in a personalized way. Academic research largely focuses on the value of recommenders for consumers, e.g., in terms of reduced information overload. To what extent and in which ways recommender systems create business value is, however, much less clear, and the literature on the topic is scattered. In this research commentary, we review existing publications on field tests of recommender systems and report which business-related performance measures were used in such real-world deployments. We summarize common challenges of measuring the business value in practice and critically discuss the value of algorithmic improvements and offline experiments as commonly done in academic environments. Overall, our review indicates that various open questions remain both regarding the realistic quantification of the business effects of recommenders and the performance assessment of recommendation algorithms in academia.