# Beyond Raw Engagement: A Counterfactual Observability Framework for Recommender Systems at Netflix

arXiv: https://arxiv.org/abs/2609.22747 · HTML: https://mlsanity.com/paper/2609.22747/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 3 of its 9 references that are in the map.
- First posted on arXiv 19 Sep 2026.

## Builds on (most cited first, 3 of 3)

- [Multisided Fairness for Recommendation](https://mlsanity.com/paper/1707.00093/) (2017)
- [AutoDebias: Learning to Debias for Recommendation](https://mlsanity.com/paper/2105.04170/) (2021)
- [Unbiased Learning for the Causal Effect of Recommendation](https://mlsanity.com/paper/2008.04563/) (2020)

## Abstract (arXiv)

Understanding the performance of large-scale recommender systems remains an underexplored challenge, especially for content creators and model developers. The raw engagement signals available to them, such as views and clicks, conflate content quality, model behavior, presentation bias, and audience reach, making it hard to attribute outcomes to the right cause. In this work, we present a general evaluation framework that enhances observability across multiple recommender systems at Netflix and demonstrate its effectiveness through several production deployments. The framework treats recommender-system observability as a counterfactual measurement problem: estimating what the recommender would have done, and what engagement would have followed, in the absence of a specific content item or model decision. We articulate three stakeholder-centered observability principles for content creators and model developers, and propose measurement methodologies covering bias reduction, relativity, and incrementality, applicable to both single-stage and cascading recommender systems and serving both audiences from a single measurement foundation.