# Personalized Chatbot Trustworthiness Ratings

arXiv: https://arxiv.org/abs/2005.10067 · HTML: https://mlsanity.com/paper/2005.10067/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 May 2020.

## Built on by (most cited first, 6 of 6)

- [Advances in Automatically Rating the Trustworthiness of Text Processing Services](https://mlsanity.com/paper/2302.09079/) (2023)
- [The Effect of Human v/s Synthetic Test Data and Round-tripping on Assessment of Sentiment Analysis Systems for Bias](https://mlsanity.com/paper/2401.12985/) (2024)
- [On Safe and Usable Chatbots for Promoting Voter Participation](https://mlsanity.com/paper/2212.11219/) (2022)
- [Evaluating Chatbots to Promote Users' Trust -- Practices and Open Problems](https://mlsanity.com/paper/2309.05680/) (2023)
- [Holistic Explainable AI (H-XAI): Extending Transparency Beyond Developers in AI-Driven Decision Making](https://mlsanity.com/paper/2508.05792/) (2025)
- [SafeChat: A Framework for Building Trustworthy Collaborative Assistants and a Case Study of its Usefulness](https://mlsanity.com/paper/2504.07995/) (2025)

## Abstract (arXiv)

Conversation agents, commonly referred to as chatbots, are increasingly deployed in many domains to allow people to have a natural interaction while trying to solve a specific problem. Given their widespread use, it is important to provide their users with methods and tools to increase users awareness of various properties of the chatbots, including non-functional properties that users may consider important in order to trust a specific chatbot. For example, users may want to use chatbots that are not biased, that do not use abusive language, that do not leak information to other users, and that respond in a style which is appropriate for the user's cognitive level. In this paper, we address the setting where a chatbot cannot be modified, its training data cannot be accessed, and yet a neutral party wants to assess and communicate its trustworthiness to a user, tailored to the user's priorities over the various trust issues. Such a rating can help users choose among alternative chatbots, developers test their systems, business leaders price their offering, and regulators set policies. We envision a personalized rating methodology for chatbots that relies on separate rating modules for each issue, and users' detected priority orderings among the relevant trust issues, to generate an aggregate personalized rating for the trustworthiness of a chatbot. The method is independent of the specific trust issues and is parametric to the aggregation procedure, thereby allowing for seamless generalization. We illustrate its general use, integrate it with a live chatbot, and evaluate it on four dialog datasets and representative user profiles, validated with user surveys.