# "Even explanations will not help in trusting [this] fundamentally biased system": A Predictive Policing Case-Study

arXiv: https://arxiv.org/abs/2504.11020 · HTML: https://mlsanity.com/paper/2504.11020/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 99 references that are in the map.
- First posted on arXiv 15 Apr 2025.

## Builds on (most cited first, 20 of 20)

- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance](https://mlsanity.com/paper/2006.14779/) (2020)
- [Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making](https://mlsanity.com/paper/2001.02114/) (2020)
- [Multimodal Explanations: Justifying Decisions and Pointing to the Evidence](https://mlsanity.com/paper/1802.08129/) (2018)
- [Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI](https://mlsanity.com/paper/2010.07487/) (2020)
- [On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection](https://mlsanity.com/paper/1811.07901/) (2018)
- [To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making](https://mlsanity.com/paper/2102.09692/) (2021)
- [Explanations Can Reduce Overreliance on AI Systems During Decision-Making](https://mlsanity.com/paper/2212.06823/) (2022)
- [Runaway Feedback Loops in Predictive Policing](https://mlsanity.com/paper/1706.09847/) (2017)
- [Explainable AI is Dead, Long Live Explainable AI! Hypothesis-driven decision support](https://mlsanity.com/paper/2302.12389/) (2023)
- [Designing for Responsible Trust in AI Systems: A Communication Perspective](https://mlsanity.com/paper/2204.13828/) (2022)
- [A systematic review and taxonomy of explanations in decision support and recommender systems](https://mlsanity.com/paper/2006.08672/) (2020)
- [Explaining Models: An Empirical Study of How Explanations Impact Fairness Judgment](https://mlsanity.com/paper/1901.07694/) (2019)
- [A Systematic Review on Fostering Appropriate Trust in Human-AI Interaction](https://mlsanity.com/paper/2311.06305/) (2023)
- ["There Is Not Enough Information": On the Effects of Explanations on Perceptions of Informational Fairness and Trustworthiness in Automated Decision-Making](https://mlsanity.com/paper/2205.05758/) (2022)
- [Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making](https://mlsanity.com/paper/2308.04375/) (2023)
- [The effect of differential victim crime reporting on predictive policing systems](https://mlsanity.com/paper/2102.00128/) (2021)
- [Should We Trust (X)AI? Design Dimensions for Structured Experimental Evaluations](https://mlsanity.com/paper/2009.06433/) (2020)

## Abstract (arXiv)

In today's society, where Artificial Intelligence (AI) has gained a vital role, concerns regarding user's trust have garnered significant attention. The use of AI systems in high-risk domains have often led users to either under-trust it, potentially causing inadequate reliance or over-trust it, resulting in over-compliance. Therefore, users must maintain an appropriate level of trust. Past research has indicated that explanations provided by AI systems can enhance user understanding of when to trust or not trust the system. However, the utility of presentation of different explanations forms still remains to be explored especially in high-risk domains. Therefore, this study explores the impact of different explanation types (text, visual, and hybrid) and user expertise (retired police officers and lay users) on establishing appropriate trust in AI-based predictive policing. While we observed that the hybrid form of explanations increased the subjective trust in AI for expert users, it did not led to better decision-making. Furthermore, no form of explanations helped build appropriate trust. The findings of our study emphasize the importance of re-evaluating the use of explanations to build [appropriate] trust in AI based systems especially when the system's use is questionable. Finally, we synthesize potential challenges and policy recommendations based on our results to design for appropriate trust in high-risk based AI-based systems.