# Algorithmic Assistance with Recommendation-Dependent Preferences

arXiv: https://arxiv.org/abs/2208.07626 · HTML: https://mlsanity.com/paper/2208.07626/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 60 references that are in the map.
- First posted on arXiv 16 Aug 2022.

## Builds on (most cited first, 17 of 17)

- ["How do I fool you?": Manipulating User Trust via Misleading Black Box Explanations](https://mlsanity.com/paper/1911.06473/) (2019)
- [The Algorithmic Automation Problem: Prediction, Triage, and Human Effort](https://mlsanity.com/paper/1903.12220/) (2019)
- [Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies](https://mlsanity.com/paper/2112.11471/) (2021)
- [Human-Algorithm Collaboration: Achieving Complementarity and Avoiding Unfairness](https://mlsanity.com/paper/2202.08821/) (2022)
- [When Humans Aren't Optimal: Robots that Collaborate with Risk-Aware Humans](https://mlsanity.com/paper/2001.04377/) (2020)
- [Who Goes First? Influences of Human-AI Workflow on Decision Making in Clinical Imaging](https://mlsanity.com/paper/2205.09696/) (2022)
- [The Challenge of Understanding What Users Want: Inconsistent Preferences and Engagement Optimization](https://mlsanity.com/paper/2202.11776/) (2022)
- [Algorithmic Risk Assessments Can Alter Human Decision-Making Processes in High-Stakes Government Contexts](https://mlsanity.com/paper/2012.05370/) (2020)
- [Experimental Evaluation of Algorithm-Assisted Human Decision-Making: Application to Pretrial Public Safety Assessment](https://mlsanity.com/paper/2012.02845/) (2020)
- [Digital Nudging with Recommender Systems: Survey and Future Directions](https://mlsanity.com/paper/2011.03413/) (2020)
- [The Conflict Between Explainable and Accountable Decision-Making Algorithms](https://mlsanity.com/paper/2205.05306/) (2022)
- [When Does Uncertainty Matter?: Understanding the Impact of Predictive Uncertainty in ML Assisted Decision Making](https://mlsanity.com/paper/2011.06167/) (2020)
- [Improving Human Sequential Decision-Making with Reinforcement Learning](https://mlsanity.com/paper/2108.08454/) (2021)
- [On the Fairness of Machine-Assisted Human Decisions](https://mlsanity.com/paper/2110.15310/) (2021)
- [Loss Aversion in Recommender Systems: Utilizing Negative User Preference to Improve Recommendation Quality](https://mlsanity.com/paper/1812.11422/) (2018)
- [Heterogeneity in Algorithm-Assisted Decision-Making: A Case Study in Child Abuse Hotline Screening](https://mlsanity.com/paper/2204.05478/) (2022)
- [Modeling Human-AI Team Decision Making](https://mlsanity.com/paper/2201.02759/) (2022)

## Built on by (most cited first, 12 of 12)

- [Human-AI Interactions and Societal Pitfalls](https://mlsanity.com/paper/2309.10448/) (2023)
- [Explaining and Improving Information Complementarities in Multi-Agent Decision-making](https://mlsanity.com/paper/2502.06152/) (2025)
- [When Are Two Lists Better than One?: Benefits and Harms in Joint Decision-making](https://mlsanity.com/paper/2308.11721/) (2023)
- [Designing Algorithmic Recommendations to Achieve Human-AI Complementarity](https://mlsanity.com/paper/2405.01484/) (2024)
- [Persuasion, Delegation, and Private Information in Algorithm-Assisted Decisions](https://mlsanity.com/paper/2402.09384/) (2024)
- [Multi-Round Human-AI Collaboration with User-Specified Requirements](https://mlsanity.com/paper/2602.17646/) (2026)
- [Explanations are a Means to an End: Decision Theoretic Explanation Evaluation](https://mlsanity.com/paper/2506.22740/) (2025)
- [Interventions Against Machine-Assisted Statistical Discrimination](https://mlsanity.com/paper/2310.04585/) (2023)
- [AI Recommendations and Non-instrumental Image Concerns](https://mlsanity.com/paper/2504.19047/) (2025)
- [Strategic Advice in the Age of Personal AI](https://mlsanity.com/paper/2603.02055/) (2026)
- [Algorithmic Feature Highlighting for Human-AI Decision-Making](https://mlsanity.com/paper/2604.22236/) (2026)
- [AI Model Design and Physician Liability](https://mlsanity.com/paper/2608.13618/) (2026)

## Abstract (arXiv)

When an algorithm provides risk assessments, we typically think of them as helpful inputs to human decisions, such as when risk scores are presented to judges or doctors. However, a decision-maker may react not only to the information provided by the algorithm. The decision-maker may also view the algorithmic recommendation as a default action, making it costly for them to deviate, such as when a judge is reluctant to overrule a high-risk assessment for a defendant or a doctor fears the consequences of deviating from recommended procedures. To address such unintended consequences of algorithmic assistance, we propose a model of joint human-machine decision-making. Within this model, we consider the effect and design of algorithmic recommendations when they affect choices not just by shifting beliefs, but also by altering preferences. We motivate this assumption from institutional factors, such as a desire to avoid audits, as well as from well-established models in behavioral science that predict loss aversion relative to a reference point. We show that recommendation-dependent preferences create inefficiencies where the decision-maker is overly responsive to the recommendation. As a remedy, we discuss algorithms that strategically withhold recommendations and show how they can improve the quality of final decisions. Concretely, we prove that an intuitive algorithm achieves minimax optimality by sending recommendations only when it is confident that their implementation would improve over an unassisted baseline decision.