mlsanityData to 14 Oct 2026

Operationalising Multi-Dimensional Evaluation for Conversational Agents: A Scalable, Governed Pipeline with Selective Re-evaluation and Model Benchmarking

arXiv 2607.12085 ↗First posted 13 Jul 2026cs.AIcited by 0

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Evaluating retail conversational agents requires methods beyond lexical-overlap metrics to assess intent alignment, factuality, helpfulness, clarity, tone, and overall response quality. Although LLM-as-a-judge methods provide scalable alternatives to human evaluation, production deployment introduces challenges in governance, reproducibility, cost, schema consistency, traceability, and reliability. We present GenAI Evaluation, a governed, configuration-driven pipeline for large-scale evaluation of retail conversational systems. It processes production chatbot logs through normalization, sharding, asynchronous execution, and schema-constrained LLM scoring. The framework evaluates helpfulness, truthfulness, clarity, tone alignment, and translation-specific dimensions. Selective re-evaluation processes only incomplete, malformed, or schema-invalid records, while schema locking, versioned configurations, validation logs, and record-level provenance support auditability. The framework processes approximately 50,000 records daily and has evaluated more than two million interactions. Validation used 12,980 stratified-random human-labeled records from four trained annotators. Classification covered 14 intents, 156 sub-intents, 18 major domains, and 129 sub-domains. The pipeline achieved a macro F1 score of 0.93 and 89% human-acceptability accuracy for translation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 12 of its 17 references that are in the map. First posted on arXiv 13 Jul 2026.

Builds on

12 of 17 references.

References · 12

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  3. 2024A Survey on LLM-as-a-Judgecited by 1,423
  4. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  5. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  6. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  7. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  8. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 218
  9. 2019Survey on Evaluation Methods for Dialogue Systemscited by 126
  10. 2017Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responsescited by 98
  11. 2024M-RewardBench: Evaluating Reward Models in Multilingual Settingscited by 50
  12. 2023A Closer Look into Automatic Evaluation Using Large Language Modelscited by 17

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026On Evaluating and Improving Conversational Agents in Production
  2. 2026Benchmarking LLM Judges for Voice-Agent Evaluation: Reliability, Calibration, and Human Oversight
  3. 2025MEDAL: A Framework for Benchmarking LLMs as Multilingual Open-Domain Dialogue Evaluators
  4. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey
  5. 2024On the Benchmarking of LLMs for Open-Domain Dialogue Evaluation
  6. 2026SAGE: A Service Agent Graph-guided Evaluation Benchmark
  7. 2026Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents
  8. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate