mlsanityData to 14 Oct 2026

Benchmarking LLM powered Chatbots: Methods and Metrics

arXiv 2308.04624 ↗First posted 8 Aug 2023cs.CL cs.AIcited by 19

Not on the map. Why?

Abstract (arXiv)

Autonomous conversational agents, i.e. chatbots, are becoming an increasingly common mechanism for enterprises to provide support to customers and partners. In order to rate chatbots, especially ones powered by Generative AI tools like Large Language Models (LLMs) we need to be able to accurately assess their performance. This is where chatbot benchmarking becomes important. In this paper, we propose the use of a novel benchmark that we call the E2E (End to End) benchmark, and show how the E2E benchmark can be used to evaluate accuracy and usefulness of the answers provided by chatbots, especially ones powered by LLMs. We evaluate an example chatbot at different levels of sophistication based on both our E2E benchmark, as well as other available metrics commonly used in the state of art, and observe that the proposed benchmark show better results compared to others. In addition, while some metrics proved to be unpredictable, the metric associated with the E2E benchmark, which uses cosine similarity performed well in evaluating chatbots. The performance of our best models shows that there are several benefits of using the cosine similarity score as a metric in the E2E benchmark.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 4 of its 6 references that are in the map. First posted on arXiv 8 Aug 2023.

Builds on

4 of 6 references.

References · 4

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  2. 2018Universal Sentence Encodercited by 489
  3. 2021Understanding the Capabilities, Limitations, and Societal Impact of Large Language Modelscited by 160
  4. 2022Modern Question Answering Datasets and Benchmarks: A Surveycited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 8; 2025: 3; 2026: 4.

19 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  2. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 57
  3. 2024Few-shot Personalization of LLMs with Mis-aligned Responsescited by 32
  4. 2023An In-depth Survey of Large Language Model-based Artificial Intelligence Agentscited by 31
  5. 2025From Prompts to Power: Measuring the Energy Footprint of LLM Inferencecited by 12
  6. 2024LLM-DaaS: LLM-driven Drone-as-a-Service Operations from Text User Requestscited by 7
  7. 2025Random-Set Large Language Modelscited by 7
  8. 2023Bridging Code Semantic and LLMs: Semantic Chain-of-Thought Prompting for Code Generationcited by 6
  9. 2024A Comparison of LLM Finetuning Methods & Evaluation Metrics with Travel Chatbot Use Casecited by 6
  10. 2024Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graphcited by 4
  11. 2023Enhancing Summarization Performance through Transformer-Based Prompt Engineering in Automated Medical Reportingcited by 3
  12. 2026TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Taskscited by 3
7 more
  1. 2024Generative AI Toolkit -- a framework for increasing the quality of LLM-based applications over their whole life cyclecited by 2
  2. 2026Reflections and New Directions for Human-Centered Large Language Modelscited by 2
  3. 2024Aligning Agents like Large Language Modelscited by 1
  4. 2024Design of a Quality Management System based on the EU Artificial Intelligence Actcited by 0
  5. 2024Is Our Chatbot Telling Lies? Assessing Correctness of an LLM-based Dutch Support Chatbotcited by 0
  6. 2026GuidaPA: Privacy-Preserving Chatbot for Public Administration via Federated Learningcited by 0
  7. 2026A Baseline Assessment for the Agentic Web Technology Stackcited by 0