# Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI

arXiv: https://arxiv.org/abs/2010.07487 · HTML: https://mlsanity.com/paper/2010.07487/

## Key facts

- Cited by 167 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Oct 2020.

## Built on by (most cited first, 30 of 167)

- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Trustworthy AI: From Principles to Practices](https://mlsanity.com/paper/2110.01167/) (2021)
- [Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond](https://mlsanity.com/paper/2109.00725/) (2021)
- [CRAFT: Concept Recursive Activation FacTorization for Explainability](https://mlsanity.com/paper/2211.10154/) (2022)
- [Reframing Human-AI Collaboration for Generating Free-Text Explanations](https://mlsanity.com/paper/2112.08674/) (2021)
- [AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap](https://mlsanity.com/paper/2306.01941/) (2023)
- [A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimation](https://mlsanity.com/paper/2306.07304/) (2023)
- [The Art of Saying No: Contextual Noncompliance in Language Models](https://mlsanity.com/paper/2407.12043/) (2024)
- [Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies](https://mlsanity.com/paper/2112.11471/) (2021)
- [Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations](https://mlsanity.com/paper/2302.02187/) (2023)
- [Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey](https://mlsanity.com/paper/2210.07700/) (2022)
- [Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimation](https://mlsanity.com/paper/2110.03051/) (2021)
- [Explainable AI is Dead, Long Live Explainable AI! Hypothesis-driven decision support](https://mlsanity.com/paper/2302.12389/) (2023)
- [Calibrating Large Language Models Using Their Generations Only](https://mlsanity.com/paper/2403.05973/) (2024)
- [Interactive Model Cards: A Human-Centered Approach to Model Documentation](https://mlsanity.com/paper/2205.02894/) (2022)
- [Uncertainty in Natural Language Generation: From Theory to Applications](https://mlsanity.com/paper/2307.15703/) (2023)
- [Determinants of LLM-assisted Decision-Making](https://mlsanity.com/paper/2402.17385/) (2024)
- [Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistant](https://mlsanity.com/paper/2502.01390/) (2025)
- [Explanation-Based Human Debugging of NLP Models: A Survey](https://mlsanity.com/paper/2104.15135/) (2021)
- [Designing for Responsible Trust in AI Systems: A Communication Perspective](https://mlsanity.com/paper/2204.13828/) (2022)
- [Trust in AI: Progress, Challenges, and Future Directions](https://mlsanity.com/paper/2403.14680/) (2024)
- [A Systematic Literature Review of User Trust in AI-Enabled Systems: An HCI Perspective](https://mlsanity.com/paper/2304.08795/) (2023)
- [Local Interpretations for Explainable Natural Language Processing: A Survey](https://mlsanity.com/paper/2103.11072/) (2021)
- [Unlocking Feature Visualization for Deeper Networks with MAgnitude Constrained Optimization](https://mlsanity.com/paper/2306.06805/) (2023)
- [Explanations, Fairness, and Appropriate Reliance in Human-AI Decision-Making](https://mlsanity.com/paper/2209.11812/) (2022)
- [Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcity](https://mlsanity.com/paper/2210.15452/) (2022)
- [Algorithm Fairness in AI for Medicine and Healthcare](https://mlsanity.com/paper/2110.00603/) (2021)
- [A Systematic Review on Fostering Appropriate Trust in Human-AI Interaction](https://mlsanity.com/paper/2311.06305/) (2023)
- [LLM-Generated Black-box Explanations Can Be Adversarially Helpful](https://mlsanity.com/paper/2405.06800/) (2024)
- [Responsible AI Pattern Catalogue: A Collection of Best Practices for AI Governance and Engineering](https://mlsanity.com/paper/2209.04963/) (2022)

## Abstract (arXiv)

Trust is a central component of the interaction between people and AI, in that 'incorrect' levels of trust may cause misuse, abuse or disuse of the technology. But what, precisely, is the nature of trust in AI? What are the prerequisites and goals of the cognitive mechanism of trust, and how can we promote them, or assess whether they are being satisfied in a given interaction? This work aims to answer these questions. We discuss a model of trust inspired by, but not identical to, sociology's interpersonal trust (i.e., trust between people). This model rests on two key properties of the vulnerability of the user and the ability to anticipate the impact of the AI model's decisions. We incorporate a formalization of 'contractual trust', such that trust between a user and an AI is trust that some implicit or explicit contract will hold, and a formalization of 'trustworthiness' (which detaches from the notion of trustworthiness in sociology), and with it concepts of 'warranted' and 'unwarranted' trust. We then present the possible causes of warranted trust as intrinsic reasoning and extrinsic behavior, and discuss how to design trustworthy AI, how to evaluate whether trust has manifested, and whether it is warranted. Finally, we elucidate the connection between trust and XAI using our formalization.