Automatic Causal Fairness Analysis with LLM-Generated Reporting

AutoML, intended as the process of automating the application of machine learning to real-world problems, is a key step for AI popularisation. Most AutoML frameworks are not accounting for the potential lack of fairness in the training data and in the corresponding predictions. We introduce textscFairMind, a software prototype aiming to automatise fairness analysis at the dataset level. We achieve that by resorting to the assumptions of the standard fairness model, recently proposed by Plevcko and Bareinboim. This allows for a sound fairness evaluation in terms of causal effects, based on counterfactual queries involving the target, possibly confounders and mediators, and the different values of an input feature we regard as protected. After the necessary data preprocessing, the tool implements a closed-form computation of the effects. LLMs are consequently exploited to generate accurate reports on the fairness levels detected in the training dataset. We achieve that in a zero-shot setup and show by examples the expected advantages with respect to a direct analysis performed by the LLM. To favour applications, extensions to ordinal protected variable and continuous targets and novel decomposition results are also discussed.