# Are Recommenders Self-Aware? Label-Free Recommendation Performance Estimation via Model Uncertainty

arXiv: https://arxiv.org/abs/2507.23208 · HTML: https://mlsanity.com/paper/2507.23208/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 42 references that are in the map.
- First posted on arXiv 31 Jul 2025.

## Builds on (most cited first, 20 of 20)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation](https://mlsanity.com/paper/2002.02126/) (2020)
- [Evidential Deep Learning to Quantify Classification Uncertainty](https://mlsanity.com/paper/1806.01768/) (2018)
- [Predictive Uncertainty Estimation via Prior Networks](https://mlsanity.com/paper/1802.10501/) (2018)
- [Revisiting the Calibration of Modern Neural Networks](https://mlsanity.com/paper/2106.07998/) (2021)
- [Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoders](https://mlsanity.com/paper/2403.03952/) (2024)
- [Variational Bayesian Last Layers](https://mlsanity.com/paper/2404.11599/) (2024)
- [SimpleX: A Simple and Strong Baseline for Collaborative Filtering](https://mlsanity.com/paper/2109.12613/) (2021)
- [Not All Relevance Scores are Equal: Efficient Uncertainty and Calibration Modeling for Deep Retrieval Models](https://mlsanity.com/paper/2105.04651/) (2021)
- [Convolutional Gaussian Embeddings for Personalized Recommendation with Uncertainty](https://mlsanity.com/paper/2006.10932/) (2020)
- [Predictive Uncertainty-based Bias Mitigation in Ranking](https://mlsanity.com/paper/2309.09833/) (2023)
- [ReChorus2.0: A Modular and Task-Flexible Recommendation Library](https://mlsanity.com/paper/2405.18058/) (2024)
- [Auto-Ensemble: An Adaptive Learning Rate Scheduling based Deep Learning Model Ensembling](https://mlsanity.com/paper/2003.11266/) (2020)
- [Uncertainty Calibration for Counterfactual Propensity Estimation in Recommendation](https://mlsanity.com/paper/2303.12973/) (2023)
- [Mitigating Exploitation Bias in Learning to Rank with an Uncertainty-aware Empirical Bayes Approach](https://mlsanity.com/paper/2305.16606/) (2023)
- [From Aggregation to Selection: User-Validated Distributed Social Recommendation](https://mlsanity.com/paper/2505.21388/) (2025)

## Abstract (arXiv)

Can a recommendation model be self-aware? This paper investigates the recommender's self-awareness by quantifying its uncertainty, which provides a label-free estimation of its performance. Such self-assessment can enable more informed understanding and decision-making before the recommender engages with any users. To this end, we propose an intuitive and effective method, probability-based List Distribution uncertainty (LiDu). LiDu measures uncertainty by determining the probability that a recommender will generate a certain ranking list based on the prediction distributions of individual items. We validate LiDu's ability to represent model self-awareness in two settings: (1) with a matrix factorization model on a synthetic dataset, and (2) with popular recommendation algorithms on real-world datasets. Experimental results show that LiDu is more correlated with recommendation performance than a series of label-free performance estimators. Additionally, LiDu provides valuable insights into the dynamic inner states of models throughout training and inference. This work establishes an empirical connection between recommendation uncertainty and performance, framing it as a step towards more transparent and self-evaluating recommender systems.