mlsanityData to 14 Oct 2026

Beyond NDCG: behavioral testing of recommender systems with RecList

arXiv 2111.09963 ↗First posted 18 Nov 2021cs.IR cs.AI cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

As with most Machine Learning systems, recommender systems are typically evaluated through performance metrics computed over held-out data points. However, real-world behavior is undoubtedly nuanced: ad hoc error analysis and deployment-specific tests must be employed to ensure the desired quality in actual deployments. In this paper, we propose RecList, a behavioral-based testing methodology. RecList organizes recommender systems by use case and introduces a general plug-and-play procedure to scale up behavioral testing. We demonstrate its capabilities by analyzing known algorithms and black-box commercial systems, and we release RecList as an open source, extensible package for the community.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 19 of its 44 references that are in the map. First posted on arXiv 18 Nov 2021.

Builds on

19 of 44 references.

References · 19

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  3. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  4. 2015Session-based Recommendations with Recurrent Neural Networkscited by 695
  5. 2019Neural Graph Collaborative Filteringcited by 661
  6. 2019A Survey on Session-based Recommender Systemscited by 97
  7. 2018Evaluation of Session-based Recommendation Algorithmscited by 37
  8. 2016Meta-Prod2Vec - Product Embeddings Using Side-Information for Recommendationcited by 30
  9. 2020M2GRL: A Multi-task Multi-view Graph Representation Learning Framework for Web-scale Recommender Systemscited by 21
  10. 2020Fantastic Embeddings and How to Align Them: Zero-Shot Inference in a Multi-Shop Scenariocited by 16
  11. 2021SIGIR 2021 E-Commerce Workshop Data Challengecited by 11
  12. 2018An Analysis of Approaches Taken in the ACM RecSys Challenge 2018 for Automatic Music Playlist Continuationcited by 9
7 more
  1. 2020BERT Goes Shopping: Comparing Distributional Models for Product Representationscited by 7
  2. 2021Transformers with multi-modal features and post-fusion context for e-commerce session-based recommendationcited by 7
  3. 2021You Do Not Need a Bigger Boat: Recommendations at Reasonable Scale in a (Mostly) Serverless and Open Stackcited by 5
  4. 2021DAG Card is the new Model Cardcited by 5
  5. 2021Recommendations and Results Organization in Netflix Searchcited by 4
  6. 2021"Are you sure?": Preliminary Insights from Scaling Product Comparisons to Multiple Shopscited by 4
  7. 2018Diversifying Music Recommendationscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 2; 2024: 0; 2025: 0; 2026: 2.

8 papers.

  1. 2023Reasonable Scale Machine Learning with Open-Source Metaflowcited by 5
  2. 2022Twitter-Demographer: A Flow-based Tool to Enrich Twitter Datacited by 1
  3. 2023Bridging Offline-Online Evaluation with a Time-dependent and Popularity Bias-free Offline Metric for Recommenderscited by 1
  4. 2022Triplet Losses-based Matrix Factorization for Robust Recommendationscited by 0
  5. 2022Track2Vec: fairness music recommendation with a GPU-free customizable-driven frameworkcited by 0
  6. 2022Item-based Variational Auto-encoder for Fair Music Recommendationcited by 0
  7. 2026SIDScope: A Diagnostic Resource for Semantic-ID Interfaces in Generative Recommendationcited by 0
  8. 2026AURA: Agentic Diagnosis and Refinement for Production Recommender Systems at Scalecited by 0