# Automated Trustworthiness Oracle Generation for Machine Learning Text Classifiers

arXiv: https://arxiv.org/abs/2410.22663 · HTML: https://mlsanity.com/paper/2410.22663/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 46 of its 86 references that are in the map.
- First posted on arXiv 30 Oct 2024.

## Builds on (most cited first, 30 of 46)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images](https://mlsanity.com/paper/1412.1897/) (2014)
- [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019)
- [HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection](https://mlsanity.com/paper/2012.10289/) (2020)
- [Unmasking Clever Hans Predictors and Assessing What Machines Really Learn](https://mlsanity.com/paper/1902.10178/) (2019)
- [Explainability for Large Language Models: A Survey](https://mlsanity.com/paper/2309.01029/) (2023)
- [DeepXplore: Automated Whitebox Testing of Deep Learning Systems](https://mlsanity.com/paper/1705.06640/) (2017)
- [Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations](https://mlsanity.com/paper/1703.03717/) (2017)
- [Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making](https://mlsanity.com/paper/2001.02114/) (2020)
- [DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars](https://mlsanity.com/paper/1708.08559/) (2017)
- [Understanding Neural Networks through Representation Erasure](https://mlsanity.com/paper/1612.08220/) (2016)
- [Machine Learning Testing: Survey, Landscapes and Horizons](https://mlsanity.com/paper/1906.10742/) (2019)
- [DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems](https://mlsanity.com/paper/1803.07519/) (2018)
- [Making deep neural networks right for the right scientific reasons by interacting with their explanations](https://mlsanity.com/paper/2001.05371/) (2020)
- [Counter-fitting Word Vectors to Linguistic Constraints](https://mlsanity.com/paper/1603.00892/) (2016)
- [AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap](https://mlsanity.com/paper/2306.01941/) (2023)
- [Shortcut Learning of Large Language Models in Natural Language Understanding](https://mlsanity.com/paper/2208.11857/) (2022)
- [Towards Improving Adversarial Training of NLP Models](https://mlsanity.com/paper/2109.00544/) (2021)
- [Towards Interpreting and Mitigating Shortcut Learning Behavior of NLU Models](https://mlsanity.com/paper/2103.06922/) (2021)
- [Software Engineering for AI-Based Systems: A Survey](https://mlsanity.com/paper/2105.01984/) (2021)
- [DLFuzz: Differential Fuzzing Testing of Deep Learning Systems](https://mlsanity.com/paper/1808.09413/) (2018)
- [Hierarchical Clustering: Objective Functions and Algorithms](https://mlsanity.com/paper/1704.02147/) (2017)
- [Explanation-Based Human Debugging of NLP Models: A Survey](https://mlsanity.com/paper/2104.15135/) (2021)
- [AttentionViz: A Global View of Transformer Attention](https://mlsanity.com/paper/2305.03210/) (2023)

## Abstract (arXiv)

Machine learning (ML) for text classification has been widely used in various domains. These applications can significantly impact ethics, economics, and human behavior, raising serious concerns about trusting ML decisions. Studies indicate that conventional metrics are insufficient to build human trust in ML models. These models often learn spurious correlations and predict based on them. In the real world, their performance can deteriorate significantly. To avoid this, a common practice is to test whether predictions are reasonable based on valid patterns in the data. Along with this, a challenge known as the trustworthiness oracle problem has been introduced. Due to the lack of automated trustworthiness oracles, the assessment requires manual validation of the decision process disclosed by explanation methods. However, this is time-consuming, error-prone, and unscalable. We propose TOKI, the first automated trustworthiness oracle generation method for text classifiers. TOKI automatically checks whether the words contributing the most to a prediction are semantically related to the predicted class. Specifically, we leverage ML explanations to extract the decision-contributing words and measure their semantic relatedness with the class based on word embeddings. We also introduce a novel adversarial attack method that targets trustworthiness vulnerabilities identified by TOKI. To evaluate their alignment with human judgement, experiments are conducted. We compare TOKI with a naive baseline based solely on model confidence and TOKI-guided adversarial attack method with A2T, a SOTA adversarial attack method. Results show that relying on prediction uncertainty cannot effectively distinguish between trustworthy and untrustworthy predictions, TOKI achieves 142% higher accuracy than the naive baseline, and TOKI-guided attack method is more effective with fewer perturbations than A2T.