# Beyond NDCG: behavioral testing of recommender systems with RecList

arXiv: https://arxiv.org/abs/2111.09963 · HTML: https://mlsanity.com/paper/2111.09963/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 44 references that are in the map.
- First posted on arXiv 18 Nov 2021.

## Builds on (most cited first, 19 of 19)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Session-based Recommendations with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06939/) (2015)
- [Neural Graph Collaborative Filtering](https://mlsanity.com/paper/1905.08108/) (2019)
- [A Survey on Session-based Recommender Systems](https://mlsanity.com/paper/1902.04864/) (2019)
- [Evaluation of Session-based Recommendation Algorithms](https://mlsanity.com/paper/1803.09587/) (2018)
- [Meta-Prod2Vec - Product Embeddings Using Side-Information for Recommendation](https://mlsanity.com/paper/1607.07326/) (2016)
- [M2GRL: A Multi-task Multi-view Graph Representation Learning Framework for Web-scale Recommender Systems](https://mlsanity.com/paper/2005.10110/) (2020)
- [Fantastic Embeddings and How to Align Them: Zero-Shot Inference in a Multi-Shop Scenario](https://mlsanity.com/paper/2007.14906/) (2020)
- [SIGIR 2021 E-Commerce Workshop Data Challenge](https://mlsanity.com/paper/2104.09423/) (2021)
- [An Analysis of Approaches Taken in the ACM RecSys Challenge 2018 for Automatic Music Playlist Continuation](https://mlsanity.com/paper/1810.01520/) (2018)
- [BERT Goes Shopping: Comparing Distributional Models for Product Representations](https://mlsanity.com/paper/2012.09807/) (2020)
- [Transformers with multi-modal features and post-fusion context for e-commerce session-based recommendation](https://mlsanity.com/paper/2107.05124/) (2021)
- [You Do Not Need a Bigger Boat: Recommendations at Reasonable Scale in a (Mostly) Serverless and Open Stack](https://mlsanity.com/paper/2107.07346/) (2021)
- [DAG Card is the new Model Card](https://mlsanity.com/paper/2110.13601/) (2021)
- [Recommendations and Results Organization in Netflix Search](https://mlsanity.com/paper/2105.14134/) (2021)
- ["Are you sure?": Preliminary Insights from Scaling Product Comparisons to Multiple Shops](https://mlsanity.com/paper/2107.03256/) (2021)
- [Diversifying Music Recommendations](https://mlsanity.com/paper/1810.01482/) (2018)

## Built on by (most cited first, 8 of 8)

- [Reasonable Scale Machine Learning with Open-Source Metaflow](https://mlsanity.com/paper/2303.11761/) (2023)
- [Twitter-Demographer: A Flow-based Tool to Enrich Twitter Data](https://mlsanity.com/paper/2201.10986/) (2022)
- [Bridging Offline-Online Evaluation with a Time-dependent and Popularity Bias-free Offline Metric for Recommenders](https://mlsanity.com/paper/2308.06885/) (2023)
- [Triplet Losses-based Matrix Factorization for Robust Recommendations](https://mlsanity.com/paper/2210.12098/) (2022)
- [Track2Vec: fairness music recommendation with a GPU-free customizable-driven framework](https://mlsanity.com/paper/2210.16590/) (2022)
- [Item-based Variational Auto-encoder for Fair Music Recommendation](https://mlsanity.com/paper/2211.01333/) (2022)
- [SIDScope: A Diagnostic Resource for Semantic-ID Interfaces in Generative Recommendation](https://mlsanity.com/paper/2608.18779/) (2026)
- [AURA: Agentic Diagnosis and Refinement for Production Recommender Systems at Scale](https://mlsanity.com/paper/2609.16625/) (2026)

## Abstract (arXiv)

As with most Machine Learning systems, recommender systems are typically evaluated through performance metrics computed over held-out data points. However, real-world behavior is undoubtedly nuanced: ad hoc error analysis and deployment-specific tests must be employed to ensure the desired quality in actual deployments. In this paper, we propose RecList, a behavioral-based testing methodology. RecList organizes recommender systems by use case and introduces a general plug-and-play procedure to scale up behavioral testing. We demonstrate its capabilities by analyzing known algorithms and black-box commercial systems, and we release RecList as an open source, extensible package for the community.