# Epistemic Trustworthiness in Generative AI: A Normative Framework for Warranted Reliance in High-Stakes Workflows

arXiv: https://arxiv.org/abs/2608.05602 · HTML: https://mlsanity.com/paper/2608.05602/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 65 references that are in the map.
- First posted on arXiv 6 Aug 2026.

## Builds on (most cited first, 20 of 20)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [AI and the Everything in the Whole Wide World Benchmark](https://mlsanity.com/paper/2111.15366/) (2021)
- [Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools](https://mlsanity.com/paper/2405.20362/) (2024)
- [Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations](https://mlsanity.com/paper/2301.07255/) (2023)
- [Gender, Race, and Intersectional Bias in Resume Screening via Language Model Retrieval](https://mlsanity.com/paper/2407.20371/) (2024)
- [Designing for Responsible Trust in AI Systems: A Communication Perspective](https://mlsanity.com/paper/2204.13828/) (2022)
- [Ignore, Trust, or Negotiate: Understanding Clinician Acceptance of AI-Based Treatment Recommendations in Health Care](https://mlsanity.com/paper/2302.00096/) (2023)
- [On the Quest for Effectiveness in Human Oversight: Interdisciplinary Perspectives](https://mlsanity.com/paper/2404.04059/) (2024)
- [Trust Me, I'm Wrong: LLMs Hallucinate with Certainty Despite Knowing the Answer](https://mlsanity.com/paper/2502.12964/) (2025)
- [The Dangers of trusting Stochastic Parrots: Faithfulness and Trust in Open-domain Conversational Question Answering](https://mlsanity.com/paper/2305.16519/) (2023)
- [Anthropic Economic Index report: Uneven geographic and enterprise AI adoption](https://mlsanity.com/paper/2511.15080/) (2025)
- [AI Ethics Principles in Practice: Perspectives of Designers and Developers](https://mlsanity.com/paper/2112.07467/) (2021)
- [Implementing Responsible AI: Tensions and Trade-Offs Between Ethics Aspects](https://mlsanity.com/paper/2304.08275/) (2023)
- [Do We Know What They Know We Know? Calibrating Student Trust in AI and Human Responses Through Mutual Theory of Mind](https://mlsanity.com/paper/2601.16960/) (2026)
- [Epistemology gives a Future to Complementarity in Human-AI Interactions](https://mlsanity.com/paper/2601.09871/) (2026)
- [Architecting Trust in Artificial Epistemic Agents](https://mlsanity.com/paper/2603.02960/) (2026)
- [The Epistemological Consequences of Large Language Models: Rethinking collective intelligence and institutional knowledge](https://mlsanity.com/paper/2512.19570/) (2025)
- [Exploring the Impact of Explainable AI and Cognitive Capabilities on Users' Decisions](https://mlsanity.com/paper/2505.01192/) (2025)
- [Learning from AVA: Early Lessons from a Curated and Trustworthy Generative AI for Policy and Development Research](https://mlsanity.com/paper/2604.17843/) (2026)

## Abstract (arXiv)

Generative AI systems are increasingly deployed in high-stakes professional contexts, where their outputs shape what users believe, how they reason, and what they treat as settled. This raises a central question for responsible AI: under what conditions is reliance on generative AI outputs epistemically warranted rather than behaviourally induced? Existing frameworks largely ask whether AI outputs are accurate, fair, explainable, safe, or trusted by users. These questions remain necessary, and each can contribute to warranted reliance. However, they do not directly specify warranted reliance as a distinct evaluative target: the conditions under which users are justified in treating AI outputs as inputs into their own reasoning. We argue that this requires an account of epistemic trustworthiness: what makes a system epistemically worthy of reliance. Drawing on philosophical accounts of trustworthiness as competence and audience-orientation, we develop a constitutive normative framework comprising three jointly necessary and non-fungible conditions. First, epistemic humility requires systems to represent and communicate the limits of their competence. Second, epistemic access requires systems to enable users to inspect, question, and contest outputs in context. Third, resistance to epistemic injustice requires systems to recognise users as legitimate epistemic agents and avoid marginalising their knowledge and experience. Through real-world case analyses in legal reasoning, medical reasoning, and hiring, we show how failures of epistemic humility, epistemic access, and resistance to epistemic injustice can produce consequential harms that standard measures of accuracy, fairness, and usability do not address on their own. We conclude by outlining design and evaluation implications for GenAI systems organised around epistemically warranted reliance rather than output correctness alone.