# What Are We Optimizing For? A Human-centric Evaluation of Deep Learning-based Movie Recommenders

arXiv: https://arxiv.org/abs/2401.11632 · HTML: https://mlsanity.com/paper/2401.11632/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 76 references that are in the map.
- First posted on arXiv 21 Jan 2024.

## Builds on (most cited first, 16 of 16)

- [Neural Collaborative Filtering](https://mlsanity.com/paper/1708.05031/) (2017)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches](https://mlsanity.com/paper/1907.06902/) (2019)
- [Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systems](https://mlsanity.com/paper/1902.05570/) (2019)
- [Managing Popularity Bias in Recommender Systems with Personalized Re-ranking](https://mlsanity.com/paper/1901.07555/) (2019)
- [BARS: Towards Open Benchmarking for Recommender Systems](https://mlsanity.com/paper/2205.09626/) (2022)
- [DaisyRec 2.0: Benchmarking Recommendation for Rigorous Evaluation](https://mlsanity.com/paper/2206.10848/) (2022)
- [Empirical Analysis of Session-Based Recommendation Algorithms](https://mlsanity.com/paper/1910.12781/) (2019)
- [Measuring "Why" in Recommender Systems: a Comprehensive Survey on the Evaluation of Explainable Recommendation](https://mlsanity.com/paper/2202.06466/) (2022)
- [Explainable Recommender Systems via Resolving Learning Representations](https://mlsanity.com/paper/2008.09316/) (2020)
- [GLocal-K: Global and Local Kernels for Recommender Systems](https://mlsanity.com/paper/2108.12184/) (2021)
- [Fairness and Transparency in Recommendation: The Users' Perspective](https://mlsanity.com/paper/2103.08786/) (2021)
- [PURS: Personalized Unexpected Recommender System for Improving User Satisfaction](https://mlsanity.com/paper/2106.02771/) (2021)
- [Personalized Re-ranking for Improving Diversity in Live Recommender Systems](https://mlsanity.com/paper/2004.06390/) (2020)
- [Interactive Content Diversity and User Exploration in Online Movie Recommenders: A Field Experiment](https://mlsanity.com/paper/2309.13296/) (2023)
- [Less Can Be More: Exploring Population Rating Dispositions with Partitioned Models in Recommender Systems](https://mlsanity.com/paper/2306.11279/) (2023)

## Abstract (arXiv)

In the past decade, deep learning (DL) models have gained prominence for their exceptional accuracy on benchmark datasets in recommender systems (RecSys). However, their evaluation has primarily relied on offline metrics, overlooking direct user perception and experience. To address this gap, we conduct a human-centric evaluation case study of four leading DL-RecSys models in the movie domain. We test how different DL-RecSys models perform in personalized recommendation generation by conducting survey study with 445 real active users. We find some DL-RecSys models to be superior in recommending novel and unexpected items and weaker in diversity, trustworthiness, transparency, accuracy, and overall user satisfaction compared to classic collaborative filtering (CF) methods. To further explain the reasons behind the underperformance, we apply a comprehensive path analysis. We discover that the lack of diversity and too much serendipity from DL models can negatively impact the consequent perceived transparency and personalization of recommendations. Such a path ultimately leads to lower summative user satisfaction. Qualitatively, we confirm with real user quotes that accuracy plus at least one other attribute is necessary to ensure a good user experience, while their demands for transparency and trust can not be neglected. Based on our findings, we discuss future human-centric DL-RecSys design and optimization strategies.