# Extracting Incentives from Black-Box Decisions

arXiv: https://arxiv.org/abs/1910.05664 · HTML: https://mlsanity.com/paper/1910.05664/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Oct 2019.

## Built on by (most cited first, 8 of 8)

- [Consequence-aware Sequential Counterfactual Generation](https://mlsanity.com/paper/2104.05592/) (2021)
- [Performative Recommendation: Diversifying Content via Strategic Incentives](https://mlsanity.com/paper/2302.04336/) (2023)
- [Personalized Algorithmic Recourse with Preference Elicitation](https://mlsanity.com/paper/2205.13743/) (2022)
- [Reinforcement Learning with Stepwise Fairness Constraints](https://mlsanity.com/paper/2211.03994/) (2022)
- [Synthesizing explainable counterfactual policies for algorithmic recourse with program synthesis](https://mlsanity.com/paper/2201.07135/) (2022)
- [Learning Actionable Counterfactual Explanations in Large State Spaces](https://mlsanity.com/paper/2404.17034/) (2024)
- [Position: Explainable AI is Causality in Disguise](https://mlsanity.com/paper/2603.28597/) (2026)
- [Shaping Human-AI Interactions to Provide Improvement Pathways and Balance Competing Objectives](https://mlsanity.com/paper/2608.05710/) (2026)

## Abstract (arXiv)

An algorithmic decision-maker incentivizes people to act in certain ways to receive better decisions. These incentives can dramatically influence subjects' behaviors and lives, and it is important that both decision-makers and decision-recipients have clarity on which actions are incentivized by the chosen model. While for linear functions, the changes a subject is incentivized to make may be clear, we prove that for many non-linear functions (e.g. neural networks, random forests), classical methods for interpreting the behavior of models (e.g. input gradients) provide poor advice to individuals on which actions they should take. In this work, we propose a mathematical framework for understanding algorithmic incentives as the challenge of solving a Markov Decision Process, where the state includes the set of input features, and the reward is a function of the model's output. We can then leverage the many toolkits for solving MDPs (e.g. tree-based planning, reinforcement learning) to identify the optimal actions each individual is incentivized to take to improve their decision under a given model. We demonstrate the utility of our method by estimating the maximally-incentivized actions in two real-world settings: a recidivism risk predictor we train using ProPublica's COMPAS dataset, and an online credit scoring tool published by the Fair Isaac Corporation (FICO).