mlsanityData to 7 Oct 2026

Does More Advice Help? The Effects of Second Opinions in AI-Assisted Decision Making

arXiv 2401.07058 ↗First posted 13 Jan 2024cs.HC cs.AIcited by 14

Not on the map. Why?

Abstract (arXiv)

AI assistance in decision-making has become popular, yet people's inappropriate reliance on AI often leads to unsatisfactory human-AI collaboration performance. In this paper, through three pre-registered, randomized human subject experiments, we explore whether and how the provision of {second opinions} may affect decision-makers' behavior and performance in AI-assisted decision-making. We find that if both the AI model's decision recommendation and a second opinion are always presented together, decision-makers reduce their over-reliance on AI while increase their under-reliance on AI, regardless whether the second opinion is generated by a peer or another AI model. However, if decision-makers have the control to decide when to solicit a peer's second opinion, we find that their active solicitations of second opinions have the potential to mitigate over-reliance on AI without inducing increased under-reliance in some cases. We conclude by discussing the implications of our findings for promoting effective human-AI collaborations in decision-making.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 33 of its 118 references that are in the map. First posted on arXiv 13 Jan 2024.

Builds on

33 of 118 references.

References · 33

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  2. 2020Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performancecited by 396
  3. 2021Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotationscited by 306
  4. 2020Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Makingcited by 281
  5. 2018Manipulating and Measuring Model Interpretabilitycited by 256
  6. 2021A Survey of Human-in-the-loop for Machine Learningcited by 214
  7. 2020Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?cited by 189
  8. 2018On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detectioncited by 159
  9. 2021To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-makingcited by 118
  10. 2022Jury Learning: Integrating Dissenting Voices into Machine Learning Modelscited by 115
  11. 2020"Why is 'Chicago' deceptive?" Towards Building Model-Driven Tutorials for Humanscited by 86
  12. 2021Towards a Science of Human-AI Decision Making: A Survey of Empirical Studiescited by 85
21 more
  1. 2023Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanationscited by 80
  2. 2019Human-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AIcited by 79
  3. 2023Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanationscited by 73
  4. 2023Who Should I Trust: AI or Myself? Leveraging Human and AI Correctness Likelihood to Promote Appropriate Trust in AI-Assisted Decision-Makingcited by 67
  5. 2021Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Makingcited by 58
  6. 2023In Search of Verifiability: Explanations Rarely Enable Complementary Performance in AI-Advised Decision Makingcited by 50
  7. 2019Quantifying Interpretability and Trust in Machine Learning Systemscited by 49
  8. 2023Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systemscited by 40
  9. 2022StoryBuddy: A Human-AI Collaborative Chatbot for Parent-Child Interactive Storytelling with Flexible Parental Involvementcited by 37
  10. 2020Are Visual Explanations Useful? A Case Study in Model-in-the-Loop Predictioncited by 36
  11. 2020The Role of Domain Expertise in User Trust and the Impact of First Impressions with Intelligent Systemscited by 34
  12. 2023Rethinking Human-AI Collaboration in Complex Medical Decision Making: A Case Study in Sepsis Diagnosiscited by 30
  13. 2022A Meta-Analysis of the Utility of Explainable Artificial Intelligence in Human-AI Decision-Makingcited by 29
  14. 2022Should I Follow AI-based Advice? Measuring Appropriate Reliance in Human-AI Decision-Makingcited by 28
  15. 2021Increasing the Speed and Accuracy of Data LabelingThrough an AI Assisted Interfacecited by 21
  16. 2021Documentation Matters: Human-Centered AI System to Assist Data Science Code Documentation in Computational Notebookscited by 17
  17. 2022A Word is Worth A Thousand Dollars: Adversarial Attack on Tweets Fools Stock Predictionscited by 15
  18. 2023Beyond Labels: Empowering Human Annotators with Natural Language Explanations through a Novel Active-Learning Architecturecited by 10
  19. 2024Decoding AI's Nudge: A Unified Framework to Predict Human Behavior in AI-assisted Decision Makingcited by 9
  20. 2023AI Reliance and Decision Quality: Fundamentals, Interdependence, and the Effects of Interventionscited by 8
  21. 2022Doubting AI Predictions: Influence-Driven Second Opinion Recommendationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 6; 2026: 5.

14 papers.

Sort:

By area:

  1. 2024Determinants of LLM-assisted Decision-Makingcited by 55
  2. 2025Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistantcited by 54
  3. 2025Is Conversational XAI All You Need? Human-AI Decision Making With a Conversational XAI Assistantcited by 26
  4. 2025As Confidence Aligns: Exploring the Effect of AI Confidence on Human Self-confidence in Human-AI Decision Makingcited by 18
  5. 2024Contrastive Explanations That Anticipate Human Misconceptions Can Improve Human Decision-Making Skillscited by 11
  6. 2025From Text to Trust: Empowering AI-assisted Decision Making with Adaptive LLM-powered Analysiscited by 9
  7. 2024Understanding the Effects of Miscalibrated AI Confidence on User Trust, Reliance, and Decision Efficacycited by 6
  8. 2025Fine-Grained Appropriate Reliance: Human-AI Collaboration with a Multi-Step Transparent Decision Workflow for Complex Task Decompositioncited by 3
  9. 2025Perils of Label Indeterminacy: A Case Study on Prediction of Neurological Recovery After Cardiac Arrestcited by 3
  10. 2026Augmenting Clinical Decision-Making with an Interactive and Interpretable AI Copilot: A Real-World User Study with Clinicians in Nephrology and Obstetricscited by 1
  11. 2026Align When They Want, Complement When They Need! Human-Centered Ensembles for Adaptive Human-AI Collaborationcited by 1
  12. 2026More Isn't Always Better: Balancing Decision Accuracy and Conformity Pressures in Multi-AI Advicecited by 1
2 more
  1. 2026Neither Replacement nor Panacea: Comparing LLM-Based Conversational and Graphical Decision Support in Industrial Taskscited by 0
  2. 2026A latent dimension of Condorcet's jury theorem for multiple AI adviserscited by 0