mlsanityData to 7 Oct 2026

Advances in Automatically Rating the Trustworthiness of Text Processing Services

arXiv 2302.09079 ↗First posted 4 Feb 2023cs.HC cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

AI services are known to have unstable behavior when subjected to changes in data, models or users. Such behaviors, whether triggered by omission or commission, lead to trust issues when AI works with humans. The current approach of assessing AI services in a black box setting, where the consumer does not have access to the AI's source code or training data, is limited. The consumer has to rely on the AI developer's documentation and trust that the system has been built as stated. Further, if the AI consumer reuses the service to build other services which they sell to their customers, the consumer is at the risk of the service providers (both data and model providers). Our approach, in this context, is inspired by the success of nutritional labeling in food industry to promote health and seeks to assess and rate AI services for trust from the perspective of an independent stakeholder. The ratings become a means to communicate the behavior of AI systems so that the consumer is informed about the risks and can make an informed decision. In this paper, we will first describe recent progress in developing rating methods for text-based machine translator AI services that have been found promising with user studies. Then, we will outline challenges and vision for a principled, multi-modal, causality-based rating methodologies and its implication for decision-support in real-world scenarios like health and food recommendation.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 23 of its 61 references that are in the map. First posted on arXiv 4 Feb 2023.

Builds on

23 of 61 references.

References · 23

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  2. 2018Datasheets for Datasetscited by 1,371
  3. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  4. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 724
  5. 2018Metrics for Explainable AI: Challenges and Prospectscited by 257
  6. 2018Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systemscited by 234
  7. 2019On instabilities of deep learning in image reconstruction - Does AI come at a cost?cited by 211
  8. 2021Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyondcited by 177
  9. 2018Assessing Gender Bias in Machine Translation -- A Case Study with Google Translatecited by 121
  10. 2017Ethical Challenges in Data-Driven Dialogue Systemscited by 86
  11. 2019Equalizing Gender Biases in Neural Machine Translation with Word Embeddings Techniquescited by 65
  12. 2018A Survey on Food Computingcited by 59
11 more
  1. 2021Deconfounded Recommendation for Alleviating Bias Amplificationcited by 51
  2. 2020Generative Interventions for Causal Learningcited by 45
  3. 2019On the Legal Compatibility of Fairness Definitionscited by 19
  4. 2021Deconfounded Causal Collaborative Filteringcited by 14
  5. 2020Bias in Data-driven AI Systems -- An Introductory Surveycited by 10
  6. 2020Personalized Chatbot Trustworthiness Ratingscited by 6
  7. 2018Towards Composable Bias Rating of AI Servicescited by 4
  8. 2017A Measure for Dialog Complexity and its Application in Streamlining Service Operationscited by 3
  9. 2022On Safe and Usable Chatbots for Promoting Voter Participationcited by 3
  10. 2022A Rich Recipe Representation as Plan to Support Expressive Multi Modal Queries on Recipe Content and Preparation Processcited by 2
  11. 2022Dynamic Causal Collaborative Filteringcited by 2

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 2; 2025: 2.

6 papers.

  1. 2023Verifying the Robustness of Automatic Credibility Assessmentcited by 8
  2. 2024The Effect of Human v/s Synthetic Test Data and Round-tripping on Assessment of Sentiment Analysis Systems for Biascited by 4
  3. 2024Rating Multi-Modal Time-Series Forecasting Models (MM-TSFM) for Robustness Through a Causal Lenscited by 2
  4. 2023Evaluating Chatbots to Promote Users' Trust -- Practices and Open Problemscited by 1
  5. 2025Holistic Explainable AI (H-XAI): Extending Transparency Beyond Developers in AI-Driven Decision Makingcited by 1
  6. 2025Creating a Causally Grounded Rating Method for Assessing the Robustness of AI Models for Time-Series Forecastingcited by 0