# Doubting AI Predictions: Influence-Driven Second Opinion Recommendation

arXiv: https://arxiv.org/abs/2205.00072 · HTML: https://mlsanity.com/paper/2205.00072/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 24 references that are in the map.
- First posted on arXiv 29 Apr 2022.

## Builds on (most cited first, 15 of 15)

- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [A Simple Baseline for Bayesian Uncertainty in Deep Learning](https://mlsanity.com/paper/1902.02476/) (2019)
- [Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance](https://mlsanity.com/paper/2006.14779/) (2020)
- [Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making](https://mlsanity.com/paper/2001.02114/) (2020)
- [Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer](https://mlsanity.com/paper/1711.06664/) (2017)
- [Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty](https://mlsanity.com/paper/2011.07586/) (2020)
- [Learning to Complement Humans](https://mlsanity.com/paper/2005.00582/) (2020)
- [A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores](https://mlsanity.com/paper/2002.08035/) (2020)
- [Direct Uncertainty Prediction for Medical Second Opinions](https://mlsanity.com/paper/1807.01771/) (2018)
- [Human-AI Collaboration with Bandit Feedback](https://mlsanity.com/paper/2105.10614/) (2021)
- [Leveraging Expert Consistency to Improve Algorithmic Decision Support](https://mlsanity.com/paper/2101.09648/) (2021)
- [When Does Uncertainty Matter?: Understanding the Impact of Predictive Uncertainty in ML Assisted Decision Making](https://mlsanity.com/paper/2011.06167/) (2020)
- [A Case for Backward Compatibility for Human-AI Teams](https://mlsanity.com/paper/1906.01148/) (2019)

## Built on by (most cited first, 2 of 2)

- [Does More Advice Help? The Effects of Second Opinions in AI-Assisted Decision Making](https://mlsanity.com/paper/2401.07058/) (2024)
- [Expanding Knowledge Graphs with Humans in the Loop](https://mlsanity.com/paper/2212.05189/) (2022)

## Abstract (arXiv)

Effective human-AI collaboration requires a system design that provides humans with meaningful ways to make sense of and critically evaluate algorithmic recommendations. In this paper, we propose a way to augment human-AI collaboration by building on a common organizational practice: identifying experts who are likely to provide complementary opinions. When machine learning algorithms are trained to predict human-generated assessments, experts' rich multitude of perspectives is frequently lost in monolithic algorithmic recommendations. The proposed approach aims to leverage productive disagreement by (1) identifying whether some experts are likely to disagree with an algorithmic assessment and, if so, (2) recommend an expert to request a second opinion from.