# Benchmarking LLM powered Chatbots: Methods and Metrics

arXiv: https://arxiv.org/abs/2308.04624 · HTML: https://mlsanity.com/paper/2308.04624/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 6 references that are in the map.
- First posted on arXiv 8 Aug 2023.

## Builds on (most cited first, 4 of 4)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Universal Sentence Encoder](https://mlsanity.com/paper/1803.11175/) (2018)
- [Understanding the Capabilities, Limitations, and Societal Impact of Large Language Models](https://mlsanity.com/paper/2102.02503/) (2021)
- [Modern Question Answering Datasets and Benchmarks: A Survey](https://mlsanity.com/paper/2206.15030/) (2022)

## Built on by (most cited first, 19 of 19)

- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey](https://mlsanity.com/paper/2503.22458/) (2025)
- [Few-shot Personalization of LLMs with Mis-aligned Responses](https://mlsanity.com/paper/2406.18678/) (2024)
- [An In-depth Survey of Large Language Model-based Artificial Intelligence Agents](https://mlsanity.com/paper/2309.14365/) (2023)
- [From Prompts to Power: Measuring the Energy Footprint of LLM Inference](https://mlsanity.com/paper/2511.05597/) (2025)
- [LLM-DaaS: LLM-driven Drone-as-a-Service Operations from Text User Requests](https://mlsanity.com/paper/2412.11672/) (2024)
- [Random-Set Large Language Models](https://mlsanity.com/paper/2504.18085/) (2025)
- [Bridging Code Semantic and LLMs: Semantic Chain-of-Thought Prompting for Code Generation](https://mlsanity.com/paper/2310.10698/) (2023)
- [A Comparison of LLM Finetuning Methods & Evaluation Metrics with Travel Chatbot Use Case](https://mlsanity.com/paper/2408.03562/) (2024)
- [Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph](https://mlsanity.com/paper/2405.02105/) (2024)
- [Enhancing Summarization Performance through Transformer-Based Prompt Engineering in Automated Medical Reporting](https://mlsanity.com/paper/2311.13274/) (2023)
- [TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks](https://mlsanity.com/paper/2605.17170/) (2026)
- [Generative AI Toolkit -- a framework for increasing the quality of LLM-based applications over their whole life cycle](https://mlsanity.com/paper/2412.14215/) (2024)
- [Reflections and New Directions for Human-Centered Large Language Models](https://mlsanity.com/paper/2605.06901/) (2026)
- [Aligning Agents like Large Language Models](https://mlsanity.com/paper/2406.04208/) (2024)
- [Design of a Quality Management System based on the EU Artificial Intelligence Act](https://mlsanity.com/paper/2408.04689/) (2024)
- [Is Our Chatbot Telling Lies? Assessing Correctness of an LLM-based Dutch Support Chatbot](https://mlsanity.com/paper/2411.00034/) (2024)
- [GuidaPA: Privacy-Preserving Chatbot for Public Administration via Federated Learning](https://mlsanity.com/paper/2606.01386/) (2026)
- [A Baseline Assessment for the Agentic Web Technology Stack](https://mlsanity.com/paper/2610.13703/) (2026)

## Abstract (arXiv)

Autonomous conversational agents, i.e. chatbots, are becoming an increasingly common mechanism for enterprises to provide support to customers and partners. In order to rate chatbots, especially ones powered by Generative AI tools like Large Language Models (LLMs) we need to be able to accurately assess their performance. This is where chatbot benchmarking becomes important. In this paper, we propose the use of a novel benchmark that we call the E2E (End to End) benchmark, and show how the E2E benchmark can be used to evaluate accuracy and usefulness of the answers provided by chatbots, especially ones powered by LLMs. We evaluate an example chatbot at different levels of sophistication based on both our E2E benchmark, as well as other available metrics commonly used in the state of art, and observe that the proposed benchmark show better results compared to others. In addition, while some metrics proved to be unpredictable, the metric associated with the E2E benchmark, which uses cosine similarity performed well in evaluating chatbots. The performance of our best models shows that there are several benefits of using the cosine similarity score as a metric in the E2E benchmark.