# FairLens: Auditing Black-box Clinical Decision Support Systems

arXiv: https://arxiv.org/abs/2011.04049 · HTML: https://mlsanity.com/paper/2011.04049/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Nov 2020.

## Built on by (most cited first, 23 of 23)

- [To trust or not to trust an explanation: using LEAF to evaluate local linear XAI methods](https://mlsanity.com/paper/2106.00461/) (2021)
- [Algorithm Fairness in AI for Medicine and Healthcare](https://mlsanity.com/paper/2110.00603/) (2021)
- [A Critical Survey on Fairness Benefits of Explainable AI](https://mlsanity.com/paper/2310.13007/) (2023)
- [A Survey on Deep Learning for Polyp Segmentation: Techniques, Challenges and Future Trends](https://mlsanity.com/paper/2311.18373/) (2023)
- [Auditing of AI: Legal, Ethical and Technical Approaches](https://mlsanity.com/paper/2407.06235/) (2024)
- [AcME -- Accelerated Model-agnostic Explanations: Fast Whitening of the Machine-Learning Black Box](https://mlsanity.com/paper/2112.12635/) (2021)
- [Algorithmic audits of algorithms, and the law](https://mlsanity.com/paper/2203.03711/) (2022)
- [Verifiable Fairness: Privacy-preserving Computation of Fairness for Machine Learning Systems](https://mlsanity.com/paper/2309.06061/) (2023)
- [Secure and Confidential Certificates of Online Fairness](https://mlsanity.com/paper/2410.02777/) (2024)
- [GRAPHSHAP: Explaining Identity-Aware Graph Classifiers Through the Language of Motifs](https://mlsanity.com/paper/2202.08815/) (2022)
- [TCM-SD: A Benchmark for Probing Syndrome Differentiation via Natural Language Processing](https://mlsanity.com/paper/2203.10839/) (2022)
- [PerFairX: Is There a Balance Between Fairness and Personality in Large Language Model Recommendations?](https://mlsanity.com/paper/2509.08829/) (2025)
- [Explainable AI is Responsible AI: How Explainability Creates Trustworthy and Socially Responsible Artificial Intelligence](https://mlsanity.com/paper/2312.01555/) (2023)
- [XAI4LLM. Let Machine Learning Models and LLMs Collaborate for Enhanced In-Context Learning in Healthcare](https://mlsanity.com/paper/2405.06270/) (2024)
- [FairLogue: A Toolkit for Intersectional Fairness Analysis in Clinical Machine Learning Models](https://mlsanity.com/paper/2604.04858/) (2026)
- [Mapping the Potential of Explainable AI for Fairness Along the AI Lifecycle](https://mlsanity.com/paper/2404.18736/) (2024)
- [Size-adaptive Hypothesis Testing for Fairness](https://mlsanity.com/paper/2506.10586/) (2025)
- [RISED: A Pre-Deployment Evaluation Framework for High-Stakes AI Decision-Support Systems, with Application to Healthcare](https://mlsanity.com/paper/2605.12895/) (2026)
- [Streamlining models with explanations in the learning loop](https://mlsanity.com/paper/2302.07760/) (2023)
- [FairComp: Workshop on Fairness and Robustness in Machine Learning for Ubiquitous Computing](https://mlsanity.com/paper/2309.12877/) (2023)
- [Fair Multivariate Adaptive Regression Splines for Ensuring Equity and Transparency](https://mlsanity.com/paper/2402.15561/) (2024)
- [FairSelect: A Systematic Evaluation of Multi-Level and Intersectional Algorithmic Fairness](https://mlsanity.com/paper/2607.08953/) (2026)
- [The Gold in Bias: Maturing the AI Design Process through Verification](https://mlsanity.com/paper/2609.29730/) (2026)

## Abstract (arXiv)

The pervasive application of algorithmic decision-making is raising concerns on the risk of unintended bias in AI systems deployed in critical settings such as healthcare. The detection and mitigation of biased models is a very delicate task which should be tackled with care and involving domain experts in the loop. In this paper we introduce FairLens, a methodology for discovering and explaining biases. We show how our tool can be used to audit a fictional commercial black-box model acting as a clinical decision support system. In this scenario, the healthcare facility experts can use FairLens on their own historical data to discover the model's biases before incorporating it into the clinical decision flow. FairLens first stratifies the available patient data according to attributes such as age, ethnicity, gender and insurance; it then assesses the model performance on such subgroups of patients identifying those in need of expert evaluation. Finally, building on recent state-of-the-art XAI (eXplainable Artificial Intelligence) techniques, FairLens explains which elements in patients' clinical history drive the model error in the selected subgroup. Therefore, FairLens allows experts to investigate whether to trust the model and to spotlight group-specific biases that might constitute potential fairness issues.