# Assessing Trustworthiness of AI Training Dataset using Subjective Logic -- A Use Case on Bias

arXiv: https://arxiv.org/abs/2508.13813 · HTML: https://mlsanity.com/paper/2508.13813/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 25 references that are in the map.
- First posted on arXiv 19 Aug 2025.

## Builds on (most cited first, 8 of 8)

- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [The Optimal Sample Complexity of PAC Learning](https://mlsanity.com/paper/1507.00473/) (2015)
- [The Dataset Multiplicity Problem: How Unreliable Data Impacts Predictions](https://mlsanity.com/paper/2304.10655/) (2023)
- [The METRIC-framework for assessing data quality for trustworthy AI in medicine: a systematic review](https://mlsanity.com/paper/2402.13635/) (2024)
- [A Deep Dive into Dataset Imbalance and Bias in Face Identification](https://mlsanity.com/paper/2203.08235/) (2022)
- [Multi-fairness under class-imbalance](https://mlsanity.com/paper/2104.13312/) (2021)
- [Quantifying Calibration Error in Neural Networks Through Evidence-Based Theory](https://mlsanity.com/paper/2411.00265/) (2024)
- [Subjective Logic Encodings](https://mlsanity.com/paper/2502.12225/) (2025)

## Built on by (most cited first, 1 of 1)

- [PaTAS: A Framework for Trust Propagation in Neural Networks Using Subjective Logic](https://mlsanity.com/paper/2511.20586/) (2025)

## Abstract (arXiv)

As AI systems increasingly rely on training data, assessing dataset trustworthiness has become critical, particularly for properties like fairness or bias that emerge at the dataset level. Prior work has used Subjective Logic to assess trustworthiness of individual data, but not to evaluate trustworthiness properties that emerge only at the level of the dataset as a whole. This paper introduces the first formal framework for assessing the trustworthiness of AI training datasets, enabling uncertainty-aware evaluations of global properties such as bias. Built on Subjective Logic, our approach supports trust propositions and quantifies uncertainty in scenarios where evidence is incomplete, distributed, and/or conflicting. We instantiate this framework on the trustworthiness property of bias, and we experimentally evaluate it based on a traffic sign recognition dataset. The results demonstrate that our method captures class imbalance and remains interpretable and robust in both centralized and federated contexts.