mlsanityData to 14 Oct 2026

Explanations, Fairness, and Appropriate Reliance in Human-AI Decision-Making

arXiv 2209.11812 ↗First posted 23 Sep 2022cs.HC cs.AIcited by 25

Not on the map. Why?

Abstract (arXiv)

In this work, we study the effects of feature-based explanations on distributive fairness of AI-assisted decisions, specifically focusing on the task of predicting occupations from short textual bios. We also investigate how any effects are mediated by humans' fairness perceptions and their reliance on AI recommendations. Our findings show that explanations influence fairness perceptions, which, in turn, relate to humans' tendency to adhere to AI recommendations. However, we see that such explanations do not enable humans to discern correct and incorrect AI recommendations. Instead, we show that they may affect reliance irrespective of the correctness of AI recommendations. Depending on which features an explanation highlights, this can foster or hinder distributive fairness: when explanations highlight features that are task-irrelevant and evidently associated with the sensitive attribute, this prompts overrides that counter AI recommendations that align with gender stereotypes. Meanwhile, if explanations appear task-relevant, this induces reliance behavior that reinforces stereotype-aligned errors. These results imply that feature-based explanations are not a reliable mechanism to improve distributive fairness.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Builds on 53 of its 136 references that are in the map. First posted on arXiv 23 Sep 2022.

Builds on

53 of 136 references.

References · 53

  1. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  2. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  3. 2016Equality of Opportunity in Supervised Learningcited by 2,007
  4. 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,552
  5. 2018A Survey Of Methods For Explaining Black Box Modelscited by 973
  6. 2017Counterfactual Fairnesscited by 804
  7. 2016Fair prediction with disparate impact: A study of bias in recidivism prediction instrumentscited by 672
  8. 2018Explaining Explanations: An Overview of Interpretability of Machine Learningcited by 508
  9. 2019Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methodscited by 443
  10. 2020Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performancecited by 399
  11. 2016Fairness Beyond Disparate Treatment & Disparate Impact: Learning Classification without Disparate Mistreatmentcited by 369
  12. 2019Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Settingcited by 349
38 more
  1. 2020Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Makingcited by 281
  2. 2018Manipulating and Measuring Model Interpretabilitycited by 256
  3. 2020Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Surveycited by 254
  4. 2019Explainable Machine Learning in Deploymentcited by 211
  5. 2018Delayed Impact of Fair Machine Learningcited by 184
  6. 2020Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AIcited by 167
  7. 2018On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detectioncited by 159
  8. 2021What Do We Want From Explainable Artificial Intelligence (XAI)? -- A Stakeholder Perspective on XAI and a Conceptual Model Guiding Interdisciplinary XAI Researchcited by 119
  9. 2021To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-makingcited by 118
  10. 2019Learning to Deceive with Attention-Based Explanationscited by 111
  11. 2019"How do I fool you?": Manipulating User Trust via Misleading Black Box Explanationscited by 111
  12. 2018'It's Reducing a Human Being to a Percentage'; Perceptions of Justice in Algorithmic Decisionscited by 91
  13. 2020Human-centered Explainable AI: Towards a Reflective Sociotechnical Approachcited by 90
  14. 2020"Why is 'Chicago' deceptive?" Towards Building Model-Driven Tutorials for Humanscited by 86
  15. 2019Fairwashing: the risk of rationalizationcited by 81
  16. 2018How do Humans Understand Explanations from Machine Learning Systems? An Evaluation of the Human-Interpretability of Explanationcited by 72
  17. 2021Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Makingcited by 58
  18. 2020Does Explainable Artificial Intelligence Improve Human Decision-Making?cited by 49
  19. 2018Stakeholders in Explainable AIcited by 48
  20. 2019What does it mean to solve the problem of discrimination in hiring? Social, technical and legal perspectives from the UK on automated hiring systemscited by 38
  21. 2021Explainability Pitfalls: Beyond Dark Patterns in Explainable AIcited by 36
  22. 2018Human Perceptions of Fairness in Algorithmic Decision Making: A Case Study of Criminal Risk Predictioncited by 35
  23. 2019Explaining Models: An Empirical Study of How Explanations Impact Fairness Judgmentcited by 31
  24. 2022A Meta-Analysis of the Utility of Explainable Artificial Intelligence in Human-AI Decision-Makingcited by 29
  25. 2021Fairness Perceptions of Algorithmic Decision-Making: A Systematic Review of the Empirical Literaturecited by 27
  26. 2020An Analysis of LIME for Text Datacited by 25
  27. 2020Making Fair ML Software using Trustworthy Explanationcited by 24
  28. 2022"There Is Not Enough Information": On the Effects of Explanations on Perceptions of Informational Fairness and Trustworthiness in Automated Decision-Makingcited by 19
  29. 2022Toward Supporting Perceptual Complementarity in Human-AI Collaboration via Reflection on Unobservablescited by 19
  30. 2022On the Influence of Explainable AI on Automation Biascited by 18
  31. 2022Algorithmic Fairness in Business Analytics: Directions for Research and Practicecited by 18
  32. 2021Auditing for Discrimination in Algorithms Delivering Job Adscited by 17
  33. 2022On the Effect of Information Asymmetry in Human-AI Teamscited by 17
  34. 2022Investigations of Performance and Bias in Human-AI Teamwork in Hiringcited by 16
  35. 2019What You See Is What You Get? The Impact of Representation Criteria on Human Bias in Hiringcited by 14
  36. 2021Diversity in Sociotechnical Machine Learning Systemscited by 13
  37. 2020Do I Look Like a Criminal? Examining how Race Presentation Impacts Human Judgement of Recidivismcited by 11
  38. 2020LimeOut: An Ensemble Approach To Improve Process Fairnesscited by 11

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 8; 2025: 7; 2026: 5.

25 papers.

Sort:

By area:

  1. 2023Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanationscited by 83
  2. 2023The Impact of Imperfect XAI on Human-AI Decision-Makingcited by 21
  3. 2023A Critical Survey on Fairness Benefits of Explainable AIcited by 16
  4. 2024Rel-A.I.: An Interaction-Centered Approach To Measuring Human-LM Reliancecited by 16
  5. 2023AI Reliance and Decision Quality: Fundamentals, Interdependence, and the Effects of Interventionscited by 9
  6. 2025From Text to Trust: Empowering AI-assisted Decision Making with Adaptive LLM-powered Analysiscited by 9
  7. 2024Don't be Fooled: The Misinformation Effect of Explanations in Human-AI Collaborationcited by 8
  8. 2024A Survey of AI Reliancecited by 5
  9. 2025Should I Share this Translation? Evaluating Quality Feedback for User Reliance on Machine Translationcited by 5
  10. 2023The Impact of Explanations on Fairness in Human-AI Decision-Making: Protected vs Proxy Featurescited by 3
  11. 2024Personalized Help for Optimizing Low-Skilled Users' Strategycited by 3
  12. 2025From Consumption to Collaboration: Measuring Interaction Patterns to Augment Human Cognition in Open-Ended Taskscited by 3
13 more
  1. 2025Perils of Label Indeterminacy: A Case Study on Prediction of Neurological Recovery After Cardiac Arrestcited by 3
  2. 2025Gender Bias in Explainability: Investigating Performance Disparity in Post-hoc Methodscited by 3
  3. 2024When Stereotypes GTG: The Impact of Predictive Text Suggestions on Gender Bias in Human-AI Co-Writingcited by 2
  4. 2024Mapping the Potential of Explainable AI for Fairness Along the AI Lifecyclecited by 1
  5. 2024Challenging the Human-in-the-loop in Algorithmic Decision-makingcited by 1
  6. 2024Overlap Number of Balls Model-Agnostic CounterFactuals (ONB-MACF): A Data-Morphology-based Counterfactual Generation Method for Trustworthy Artificial Intelligencecited by 1
  7. 2025Let Them Down Easy! Contextual Effects of LLM Guardrails on User Perceptions and Preferencescited by 1
  8. 2026Effects of Generative AI Errors on User Reliance Across Task Difficultycited by 1
  9. 2026A Framework for Measuring Appropriate Reliance on Set-Valued AI Advicecited by 1
  10. 2025What Do People Want to Know About Artificial Intelligence (AI)? The Importance of Answering End-User Questions to Explain Autonomous Vehicle (AV) Decisionscited by 0
  11. 2026Transferable XAI: Relating Understanding Across Domains with Explanation Transfercited by 0
  12. 2026Power Echoes: Investigating Moderation Biases in Online Power-Asymmetric Conflictscited by 0
  13. 2026SketchXplain: Intuitive Visual Explanations of Image Classifiers with Sketchescited by 0