How to Dogfood Your AI Chat Agent: A Three-Layer Evaluation Framework with Goal-Directed NPC Simulation
On the map: LLM agents and tool use → Social simulation and multi-agent debate
Abstract (arXiv)
Production teams deploying LLM chat agents face a specific quality assurance gap: existing evaluation tools test individual responses or simulate social interactions, but none systematically verify whether real users can achieve their goals through multi-turn conversation. We introduce a three-layer dogfooding framework that bridges this gap by combining canonical question-bank testing (Layer 1), random-walk multi-turn evaluation (Layer 2), and a goal-directed NPC (Non-Player Character) simulator with five structured goal types and a ten-category failure taxonomy (Layer 3). In a longitudinal case study on a production multi-agent system over roughly three months (257 evaluation runs; a 108-scenario NPC suite), we find that the three layers produce complementary regression signals: cross-layer correlation for response quality is weak within a synchronized run (Spearman rho between -0.15 and 0.14) and negative across the longitudinal series (rho down to -0.46), confirming that canonical correctness does not predict goal-directed conversation success. The NPC simulator achieves 77 percent goal achievement at 0.17 dollars per run (6,272x cheaper than human evaluation), enabling daily CI/CD integration with automated PROMOTE/HOLD/ROLLBACK release decisions. We release full prompt templates, the failure taxonomy, and a Python-first replicability guide so that other teams can adopt the framework for their own LLM chat agents.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 12 of its 33 references that are in the map. First posted on arXiv 8 Jun 2026.
Builds on
12 of 33 references.
References · 12
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2020Towards a Human-like Open-Domain Chatbotcited by 467
- 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 350
- 2017Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responsescited by 98
- 2024LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Textscited by 81
- 2025SimulatorArena: Are User Simulators Reliable Proxies for Multi-Turn Evaluation of AI Assistants?cited by 45
- 2026RubricEval: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Followingcited by 19
- 2025When AIs Judge AIs: The Rise of Agent-as-a-Judge Evaluation for LLMscited by 16
- 2026JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systemscited by 10
- 2025Mind the Metrics: Patterns for Telemetry-Aware In-IDE AI Application Development using the Model Context Protocol (MCP)cited by 4
- 2026CollabEval: Enhancing LLM-as-a-Judge via Multi-Agent Collaborationcited by 4
- 2026Automated Self-Testing as a Quality Gate: Evidence-Driven Release Management for LLM Applicationscited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Mind the Goal: Data-Efficient Goal-Oriented Evaluation of Conversational Agents and Chatbots using Teacher Models
- 2025Agentic Persona Control and Task State Tracking for Realistic User Simulation in Interactive Scenarios
- 2025Simulating Persuasive Dialogues on Meat Reduction with Generative Agents
- 2025Goal Alignment in LLM-Based User Simulators for Conversational AI
- 2026Operationalising Multi-Dimensional Evaluation for Conversational Agents: A Scalable, Governed Pipeline with Selective Re-evaluation and Model Benchmarking
- 2025The Behavior Gap: Evaluating Zero-shot LLM Agents in Complex Task-Oriented Dialogs
- 2026S3Gym: Can LLMs Turn Self-Testing and Self-Judging into Self-Improvement?
- 2025The Social Laboratory: A Psychometric Framework for Multi-Agent LLM Evaluation