mlsanityData to 14 Oct 2026

End-to-End Chatbot Evaluation with Adaptive Reasoning and Uncertainty Filtering

arXiv 2603.10570 ↗First posted 11 Mar 2026cs.CLcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

also in 1 more
Abstract (arXiv)

Large language models (LLMs) combined with retrieval augmented generation have enabled the deployment of domain-specific chatbots, but these systems remain prone to generating unsupported or incorrect answers. Reliable evaluation is therefore critical, yet manual review is costly and existing frameworks often depend on curated test sets and static metrics, limiting scalability. We propose an end-to-end automatic evaluator designed to substantially reduce human effort. Our system generates Q&A pairs directly from the underlying knowledge base, uses LLMs to judge chatbot responses against reference answers, and applies confidence-based filtering to highlight uncertain cases. Applied to a Vietnamese news dataset, the evaluator achieves high agreement with human judgments while significantly lowering review overhead. The framework is modular and language-agnostic, making it readily adaptable to diverse domains. This work introduces a practical, scalable solution for evaluating chatbots with minimal reliance on manual intervention.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 10 of its 15 references that are in the map. First posted on arXiv 11 Mar 2026.

Builds on

10 of 15 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization22 May

Other references · 9

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  3. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 452
  4. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 418
  5. 2023Self-Evaluation Improves Selective Generation in Large Language Modelscited by 76
  6. 2023Adaptation with Self-Evaluation to Improve Selective Prediction in LLMscited by 50
  7. 2024Can LLM Generate Culturally Relevant Commonsense QA Data? Case Study in Indonesian and Sundanesecited by 36
  8. 2024SciQAG: A Framework for Auto-Generated Science Question Answering Dataset with Fine-grained Evaluationcited by 24
  9. 2023Dialogizer: Context-aware Conversational-QA Dataset Generation from Textual Sourcescited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023A Self-enhancement Approach for Domain-specific Chatbot Training via Knowledge Mining and Digest
  2. 2024Evaluating the Retrieval Component in LLM-Based Question Answering Systems
  3. 2025MEDAL: A Framework for Benchmarking LLMs as Multilingual Open-Domain Dialogue Evaluators
  4. 2025A Knowledge Graph and a Tripartite Evaluation Framework Make Retrieval-Augmented Generation Scalable and Transparent
  5. 2024Towards Optimizing and Evaluating a Retrieval Augmented QA Chatbot using LLMs with Human in the Loop
  6. 2024On the Benchmarking of LLMs for Open-Domain Dialogue Evaluation
  7. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate
  8. 2026Operationalising Multi-Dimensional Evaluation for Conversational Agents: A Scalable, Governed Pipeline with Selective Re-evaluation and Model Benchmarking