# Against Algorithmic Exploitation of Human Vulnerabilities

arXiv: https://arxiv.org/abs/2301.04993 · HTML: https://mlsanity.com/paper/2301.04993/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 70 references that are in the map.
- First posted on arXiv 12 Jan 2023.

## Builds on (most cited first, 14 of 14)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Bias](https://mlsanity.com/paper/1810.01943/) (2018)
- [Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy](https://mlsanity.com/paper/2002.04087/) (2020)
- [Acquisition of Chess Knowledge in AlphaZero](https://mlsanity.com/paper/2111.09259/) (2021)
- [Depression and Self-Harm Risk Assessment in Online Forums](https://mlsanity.com/paper/1709.01848/) (2017)
- [What Makes a Dark Pattern... Dark? Design Attributes, Normative Considerations, and Measurement Methods](https://mlsanity.com/paper/2101.04843/) (2021)
- [Predicting Demographics, Moral Foundations, and Human Values from Digital Behaviors](https://mlsanity.com/paper/1712.01930/) (2017)
- [Digital Voodoo Dolls](https://mlsanity.com/paper/2105.02738/) (2021)
- [You Are What Apps You Use: Demographic Prediction Based on User's Apps](https://mlsanity.com/paper/1603.00059/) (2016)
- [Estimating individual employment status using mobile phone network data](https://mlsanity.com/paper/1612.03870/) (2016)
- [Modeling patterns of smartphone usage and their relationship to cognitive health](https://mlsanity.com/paper/1911.05683/) (2019)

## Abstract (arXiv)

Decisions such as which movie to watch next, which song to listen to, or which product to buy online, are increasingly influenced by recommender systems and user models that incorporate information on users' past behaviours, preferences, and digitally created content. Machine learning models that enable recommendations and that are trained on user data may unintentionally leverage information on human characteristics that are considered vulnerabilities, such as depression, young age, or gambling addiction. The use of algorithmic decisions based on latent vulnerable state representations could be considered manipulative and could have a deteriorating impact on the condition of vulnerable individuals. In this paper, we are concerned with the problem of machine learning models inadvertently modelling vulnerabilities, and want to raise awareness for this issue to be considered in legislation and AI ethics. Hence, we define and describe common vulnerabilities, and illustrate cases where they are likely to play a role in algorithmic decision-making. We propose a set of requirements for methods to detect the potential for vulnerability modelling, detect whether vulnerable groups are treated differently by a model, and detect whether a model has created an internal representation of vulnerability. We conclude that explainable artificial intelligence methods may be necessary for detecting vulnerability exploitation by machine learning-based recommendation systems.