# A Proposal for Evaluating the Operational Risk for ChatBots based on Large Language Models

arXiv: https://arxiv.org/abs/2505.04784 · HTML: https://mlsanity.com/paper/2505.04784/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 25 references that are in the map.
- First posted on arXiv 7 May 2025.

## Builds on (most cited first, 5 of 5)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [FIRST: A Framework for Optimizing Information Quality in Mobile Crowdsensing Systems](https://mlsanity.com/paper/1804.11147/) (2018)
- [A Literature Survey of Recent Advances in Chatbots](https://mlsanity.com/paper/2201.06657/) (2022)

## Built on by (most cited first, 1 of 1)

- [Get Experience from Practice: LLM Agents with Record & Replay](https://mlsanity.com/paper/2505.17716/) (2025)

## Abstract (arXiv)

The emergence of Generative AI (Gen AI) and Large Language Models (LLMs) has enabled more advanced chatbots capable of human-like interactions. However, these conversational agents introduce a broader set of operational risks that extend beyond traditional cybersecurity considerations. In this work, we propose a novel, instrumented risk-assessment metric that simultaneously evaluates potential threats to three key stakeholders: the service-providing organization, end users, and third parties. Our approach incorporates the technical complexity required to induce erroneous behaviors in the chatbot--ranging from non-induced failures to advanced prompt-injection attacks--as well as contextual factors such as the target industry, user age range, and vulnerability severity. To validate our metric, we leverage Garak, an open-source framework for LLM vulnerability testing. We further enhance Garak to capture a variety of threat vectors (e.g., misinformation, code hallucinations, social engineering, and malicious code generation). Our methodology is demonstrated in a scenario involving chatbots that employ retrieval-augmented generation (RAG), showing how the aggregated risk scores guide both short-term mitigation and longer-term improvements in model design and deployment. The results underscore the importance of multi-dimensional risk assessments in operationalizing secure, reliable AI-driven conversational systems.