# A Unified Causal Framework for Auditing Recommender Systems for Ethical Concerns

arXiv: https://arxiv.org/abs/2409.13210 · HTML: https://mlsanity.com/paper/2409.13210/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 37 references that are in the map.
- First posted on arXiv 20 Sep 2024.

## Builds on (most cited first, 18 of 18)

- [Counterfactual Fairness](https://mlsanity.com/paper/1703.06856/) (2017)
- [Fine-Tuning Language Models with Just Forward Passes](https://mlsanity.com/paper/2305.17333/) (2023)
- [Recommendations as Treatments: Debiasing Learning and Evaluation](https://mlsanity.com/paper/1602.05352/) (2016)
- [Bias and Debias in Recommender System: A Survey and Future Directions](https://mlsanity.com/paper/2010.03240/) (2020)
- [Counterfactual Explanations and Algorithmic Recourses for Machine Learning: A Review](https://mlsanity.com/paper/2010.10596/) (2020)
- [How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utility](https://mlsanity.com/paper/1710.11214/) (2017)
- [Managing Popularity Bias in Recommender Systems with Personalized Re-ranking](https://mlsanity.com/paper/1901.07555/) (2019)
- [Counterfactual Risk Assessments, Evaluation, and Fairness](https://mlsanity.com/paper/1909.00066/) (2019)
- [The Use and Misuse of Counterfactuals in Ethical Machine Learning](https://mlsanity.com/paper/2102.05085/) (2021)
- [Causal Conceptions of Fairness and their Consequences](https://mlsanity.com/paper/2207.05302/) (2022)
- [Fighting Fire with Fire: Using Antidote Data to Improve Polarization and Fairness of Recommender Systems](https://mlsanity.com/paper/1812.01504/) (2018)
- [Estimating the Causal Impact of Recommendation Systems from Observational Data](https://mlsanity.com/paper/1510.05569/) (2015)
- [Recommendations and User Agency: The Reachability of Collaboratively-Filtered Information](https://mlsanity.com/paper/1912.10068/) (2019)
- [Fair ranking: a critical review, challenges, and future directions](https://mlsanity.com/paper/2201.12662/) (2022)
- [Measuring Recommender System Effects with Simulated Users](https://mlsanity.com/paper/2101.04526/) (2021)
- [Unbiased Learning for the Causal Effect of Recommendation](https://mlsanity.com/paper/2008.04563/) (2020)
- [Long-term Dynamics of Fairness Intervention in Connection Recommender Systems](https://mlsanity.com/paper/2203.16432/) (2022)
- [Quantifying Availability and Discovery in Recommender Systems via Stochastic Reachability](https://mlsanity.com/paper/2107.00833/) (2021)

## Built on by (most cited first, 1 of 1)

- [Gaming the Metric, Not the Harm: Certifying Safety Audits against Strategic Platform Manipulation](https://mlsanity.com/paper/2605.06324/) (2026)

## Abstract (arXiv)

As recommender systems become widely deployed in different domains, they increasingly influence their users' beliefs and preferences. Auditing recommender systems is crucial as it not only ensures the continuous improvement of recommendation algorithms but also safeguards against potential issues like biases and ethical concerns. In this paper, we view recommender system auditing from a causal lens and provide a general recipe for defining auditing metrics. Under this general causal auditing framework, we categorize existing auditing metrics and identify gaps in them -- notably, the lack of metrics for auditing user agency while accounting for the multi-step dynamics of the recommendation process. We leverage our framework and propose two classes of such metrics:future- and past-reacheability and stability, that measure the ability of a user to influence their own and other users' recommendations, respectively. We provide both a gradient-based and a black-box approach for computing these metrics, allowing the auditor to compute them under different levels of access to the recommender system. In our experiments, we demonstrate the efficacy of methods for computing the proposed metrics and inspect the design of recommender systems through these proposed metrics.