# Advances in Automatically Rating the Trustworthiness of Text Processing Services

arXiv: https://arxiv.org/abs/2302.09079 · HTML: https://mlsanity.com/paper/2302.09079/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 61 references that are in the map.
- First posted on arXiv 4 Feb 2023.

## Builds on (most cited first, 23 of 23)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [Metrics for Explainable AI: Challenges and Prospects](https://mlsanity.com/paper/1812.04608/) (2018)
- [Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems](https://mlsanity.com/paper/1805.04508/) (2018)
- [On instabilities of deep learning in image reconstruction - Does AI come at a cost?](https://mlsanity.com/paper/1902.05300/) (2019)
- [Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond](https://mlsanity.com/paper/2109.00725/) (2021)
- [Assessing Gender Bias in Machine Translation -- A Case Study with Google Translate](https://mlsanity.com/paper/1809.02208/) (2018)
- [Ethical Challenges in Data-Driven Dialogue Systems](https://mlsanity.com/paper/1711.09050/) (2017)
- [Equalizing Gender Biases in Neural Machine Translation with Word Embeddings Techniques](https://mlsanity.com/paper/1901.03116/) (2019)
- [A Survey on Food Computing](https://mlsanity.com/paper/1808.07202/) (2018)
- [Deconfounded Recommendation for Alleviating Bias Amplification](https://mlsanity.com/paper/2105.10648/) (2021)
- [Generative Interventions for Causal Learning](https://mlsanity.com/paper/2012.12265/) (2020)
- [On the Legal Compatibility of Fairness Definitions](https://mlsanity.com/paper/1912.00761/) (2019)
- [Deconfounded Causal Collaborative Filtering](https://mlsanity.com/paper/2110.07122/) (2021)
- [Bias in Data-driven AI Systems -- An Introductory Survey](https://mlsanity.com/paper/2001.09762/) (2020)
- [Personalized Chatbot Trustworthiness Ratings](https://mlsanity.com/paper/2005.10067/) (2020)
- [Towards Composable Bias Rating of AI Services](https://mlsanity.com/paper/1808.00089/) (2018)
- [A Measure for Dialog Complexity and its Application in Streamlining Service Operations](https://mlsanity.com/paper/1708.04134/) (2017)
- [On Safe and Usable Chatbots for Promoting Voter Participation](https://mlsanity.com/paper/2212.11219/) (2022)
- [A Rich Recipe Representation as Plan to Support Expressive Multi Modal Queries on Recipe Content and Preparation Process](https://mlsanity.com/paper/2203.17109/) (2022)
- [Dynamic Causal Collaborative Filtering](https://mlsanity.com/paper/2208.11094/) (2022)

## Built on by (most cited first, 6 of 6)

- [Verifying the Robustness of Automatic Credibility Assessment](https://mlsanity.com/paper/2303.08032/) (2023)
- [The Effect of Human v/s Synthetic Test Data and Round-tripping on Assessment of Sentiment Analysis Systems for Bias](https://mlsanity.com/paper/2401.12985/) (2024)
- [Rating Multi-Modal Time-Series Forecasting Models (MM-TSFM) for Robustness Through a Causal Lens](https://mlsanity.com/paper/2406.12908/) (2024)
- [Evaluating Chatbots to Promote Users' Trust -- Practices and Open Problems](https://mlsanity.com/paper/2309.05680/) (2023)
- [Holistic Explainable AI (H-XAI): Extending Transparency Beyond Developers in AI-Driven Decision Making](https://mlsanity.com/paper/2508.05792/) (2025)
- [Creating a Causally Grounded Rating Method for Assessing the Robustness of AI Models for Time-Series Forecasting](https://mlsanity.com/paper/2502.12226/) (2025)

## Abstract (arXiv)

AI services are known to have unstable behavior when subjected to changes in data, models or users. Such behaviors, whether triggered by omission or commission, lead to trust issues when AI works with humans. The current approach of assessing AI services in a black box setting, where the consumer does not have access to the AI's source code or training data, is limited. The consumer has to rely on the AI developer's documentation and trust that the system has been built as stated. Further, if the AI consumer reuses the service to build other services which they sell to their customers, the consumer is at the risk of the service providers (both data and model providers). Our approach, in this context, is inspired by the success of nutritional labeling in food industry to promote health and seeks to assess and rate AI services for trust from the perspective of an independent stakeholder. The ratings become a means to communicate the behavior of AI systems so that the consumer is informed about the risks and can make an informed decision. In this paper, we will first describe recent progress in developing rating methods for text-based machine translator AI services that have been found promising with user studies. Then, we will outline challenges and vision for a principled, multi-modal, causality-based rating methodologies and its implication for decision-support in real-world scenarios like health and food recommendation.