# We're Still Doing It (All) Wrong: Recommender Systems, Fifteen Years Later

arXiv: https://arxiv.org/abs/2509.09414 · HTML: https://mlsanity.com/paper/2509.09414/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 59 references that are in the map.
- First posted on arXiv 11 Sep 2025.

## Builds on (most cited first, 22 of 22)

- [How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utility](https://mlsanity.com/paper/1710.11214/) (2017)
- [Troubling Trends in Machine Learning Scholarship](https://mlsanity.com/paper/1807.03341/) (2018)
- [Sequence-Aware Recommender Systems](https://mlsanity.com/paper/1802.08452/) (2018)
- [Exploring Data Splitting Strategies for the Evaluation of Recommendation Models](https://mlsanity.com/paper/2007.13237/) (2020)
- [Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs](https://mlsanity.com/paper/2103.06076/) (2021)
- [Evaluating the Robustness of Off-Policy Evaluation](https://mlsanity.com/paper/2108.13703/) (2021)
- [Building Human Values into Recommender Systems: An Interdisciplinary Synthesis](https://mlsanity.com/paper/2207.10192/) (2022)
- [Widespread Flaws in Offline Evaluation of Recommender Systems](https://mlsanity.com/paper/2307.14951/) (2023)
- [Trustworthy Recommender Systems](https://mlsanity.com/paper/2208.06265/) (2022)
- [T-RECS: A Simulation Tool to Study the Societal Impact of Recommender Systems](https://mlsanity.com/paper/2107.08959/) (2021)
- [Do LLMs Memorize Recommendation Datasets? A Preliminary Study on MovieLens-1M](https://mlsanity.com/paper/2505.10212/) (2025)
- [From Clicks to Carbon: The Environmental Toll of Recommender Systems](https://mlsanity.com/paper/2408.08203/) (2024)
- [Recommender Systems for Sustainability: Overview and Research Issues](https://mlsanity.com/paper/2412.03620/) (2024)
- [Search and Society: Reimagining Information Access for Radical Futures](https://mlsanity.com/paper/2403.17901/) (2024)
- [Can We Trust Recommender System Fairness Evaluation? The Role of Fairness and Relevance](https://mlsanity.com/paper/2405.18276/) (2024)
- [De-centering the (Traditional) User: Multistakeholder Evaluation of Recommender Systems](https://mlsanity.com/paper/2501.05170/) (2025)
- [Recommending With, Not For: Co-Designing Recommender Systems for Social Good](https://mlsanity.com/paper/2508.03792/) (2025)
- [Distributionally-Informed Recommender System Evaluation](https://mlsanity.com/paper/2309.05892/) (2023)
- [Evaluating the performance-deviation of itemKNN in RecBole and LensKit](https://mlsanity.com/paper/2407.13531/) (2024)
- [Recommender Systems for Good (RS4Good): Survey of Use Cases and a Call to Action for Research that Matters](https://mlsanity.com/paper/2411.16645/) (2024)
- [Recommender Systems for Social Good: The Role of Accountability and Sustainability](https://mlsanity.com/paper/2501.05964/) (2025)
- [Extending MovieLens-32M to Provide New Evaluation Objectives](https://mlsanity.com/paper/2504.01863/) (2025)

## Built on by (most cited first, 1 of 1)

- [Don't Waste It: Guiding Generative Recommenders with Structured Human Priors via Multi-Head Decoding](https://mlsanity.com/paper/2511.10492/) (2025)

## Abstract (arXiv)

In 2011, Xavier Amatriain sounded the alarm: recommender systems research was "doing it all wrong" [1]. His critique, rooted in statistical misinterpretation and methodological shortcuts, remains as relevant today as it was then. But rather than correcting course, we added new layers of sophistication on top of the same broken foundations. This paper revisits Amatriain's diagnosis and argues that many of the conceptual, epistemological, and infrastructural failures he identified still persist, in more subtle or systemic forms. Drawing on recent work in reproducibility, evaluation methodology, environmental impact, and participatory design, we showcase how the field's accelerating complexity has outpaced its introspection. We highlight ongoing community-led initiatives that attempt to shift the paradigm, including workshops, evaluation frameworks, and calls for value-sensitive and participatory research. At the same time, we contend that meaningful change will require not only new metrics or better tooling, but a fundamental reframing of what recommender systems research is for, who it serves, and how knowledge is produced and validated. Our call is not just for technical reform, but for a recommender systems research agenda grounded in epistemic humility, human impact, and sustainable practice.