# An Empirical Study of Accuracy, Fairness, Explainability, Distributional Robustness, and Adversarial Robustness

arXiv: https://arxiv.org/abs/2109.14653 · HTML: https://mlsanity.com/paper/2109.14653/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 28 references that are in the map.
- First posted on arXiv 29 Sep 2021.

## Builds on (most cited first, 13 of 13)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [DeepFool: a simple and accurate method to fool deep neural networks](https://mlsanity.com/paper/1511.04599/) (2015)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Certifying and removing disparate impact](https://mlsanity.com/paper/1412.3756/) (2014)
- [Towards Robust Interpretability with Self-Explaining Neural Networks](https://mlsanity.com/paper/1806.07538/) (2018)
- [HopSkipJumpAttack: A Query-Efficient Decision-Based Attack](https://mlsanity.com/paper/1904.02144/) (2019)
- [AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Bias](https://mlsanity.com/paper/1810.01943/) (2018)
- [Adversarial Robustness Toolbox v1.0.0](https://mlsanity.com/paper/1807.01069/) (2018)
- [One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques](https://mlsanity.com/paper/1909.03012/) (2019)
- [Is Robustness the Cost of Accuracy? -- A Comprehensive Study on the Robustness of 18 Deep Image Classification Models](https://mlsanity.com/paper/1808.01688/) (2018)
- [Active Fairness in Algorithmic Decision Making](https://mlsanity.com/paper/1810.00031/) (2018)
- [Understanding racial bias in health using the Medical Expenditure Panel Survey data](https://mlsanity.com/paper/1911.01509/) (2019)

## Built on by (most cited first, 4 of 4)

- [An Empirical Study of Modular Bias Mitigators and Ensembles](https://mlsanity.com/paper/2202.00751/) (2022)
- [Data Optimization in Deep Learning: A Survey](https://mlsanity.com/paper/2310.16499/) (2023)
- [Navigating Ensemble Configurations for Algorithmic Fairness](https://mlsanity.com/paper/2210.05594/) (2022)
- [Function Composition in Trustworthy Machine Learning: Implementation Choices, Insights, and Questions](https://mlsanity.com/paper/2302.09190/) (2023)

## Abstract (arXiv)

To ensure trust in AI models, it is becoming increasingly apparent that evaluation of models must be extended beyond traditional performance metrics, like accuracy, to other dimensions, such as fairness, explainability, adversarial robustness, and distribution shift. We describe an empirical study to evaluate multiple model types on various metrics along these dimensions on several datasets. Our results show that no particular model type performs well on all dimensions, and demonstrate the kinds of trade-offs involved in selecting models evaluated along multiple dimensions.