mlsanityData to 14 Oct 2026

Agent-Testing Agent: A Meta-Agent for Automated Testing and Evaluation of Conversational AI Agents

arXiv 2508.17393 ↗First posted 24 Aug 2025cs.CL cs.AIcited by 2

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

LLM agents are increasingly deployed to plan, retrieve, and write with tools, yet evaluation still leans on static benchmarks and small human studies. We present the Agent-Testing Agent (ATA), a meta-agent that combines static code analysis, designer interrogation, literature mining, and persona-driven adversarial test generation whose difficulty adapts via judge feedback. Each dialogue is scored with an LLM-as-a-Judge (LAAJ) rubric and used to steer subsequent tests toward the agent's weakest capabilities. On a travel planner and a Wikipedia writer, the ATA surfaces more diverse and severe failures than expert annotators while matching severity, and finishes in 20--30 minutes versus ten-annotator rounds that took days. Ablating code analysis and web search increases variance and miscalibration, underscoring the value of evidence-grounded test generation. The ATA outputs quantitative metrics and qualitative bug reports for developers. We release the full methodology and open-source implementation for reproducible agent testing: https://github.com/KhalilMrini/Agent-Testing-Agent

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 11 of its 12 references that are in the map. First posted on arXiv 24 Aug 2025.

Builds on

11 of 12 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025IntellAgent: A Multi-Agent Framework for Evaluating Conversational AI Systems19 Jan
  2. 2024Automated test generation to evaluate tool-augmented LLMs as conversational AI agents24 Sep
  3. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agents2 Feb

Other references · 8

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2024Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Modelscited by 154
  4. 2025JudgeLRM: Large Reasoning Models as a Judgecited by 78
  5. 2024Large Language Models Can Solve Real-World Planning Rigorously with Formal Verification Toolscited by 47
  6. 2024Automated Red Teaming with GOAT: the Generative Offensive Agent Testercited by 45
  7. 2025FACT-AUDIT: An Adaptive Multi-Agent Framework for Dynamic Fact-Checking Evaluation of Large Language Modelscited by 19
  8. 2025MAEBE: Multi-Agent Emergent Behavior Frameworkcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026DynSTEER: Dynamic Stage-wise Trajectory Evaluation and Execution-time Review for Agentscited by 1
  2. 2026Kozuchi Agent: A Language-Agnostic Open-Weight Agent for Software Repaircited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Configurable multi-agent framework for scalable and realistic testing of llm-based agents
  2. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey
  3. 2026Talk, Evaluate, Diagnose: User-aware Agent Evaluation with Automated Error Analysis
  4. 2024AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents
  5. 2026Agent-as-a-Judge
  6. 2024Agent-as-a-Judge: Evaluate Agents with Agents
  7. 2026AgentSim: A Platform for Verifiable Agent-Trace Simulation
  8. 2025The Social Laboratory: A Psychometric Framework for Multi-Agent LLM Evaluation