# AgenticEval: Toward Agentic and Self-Evolving Safety Evaluation of Large Language Models

arXiv: https://arxiv.org/abs/2509.26100 · HTML: https://mlsanity.com/paper/2509.26100/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 12 of its 23 references that are in the map.
- First posted on arXiv 30 Sep 2025.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)
- Synthetic data and self-improvement for LLMs / [Safety of self-evolving systems](https://mlsanity.com/synth/safety-of-self-evolving-systems/)

## Builds on (most cited first, 12 of 12)

- [DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models](https://mlsanity.com/paper/2306.11698/) (2023)
- [AI safety via debate](https://mlsanity.com/paper/1805.00899/) (2018)
- [PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Finance](https://mlsanity.com/paper/2306.05443/) (2023)
- [AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMs](https://mlsanity.com/paper/2410.05295/) (2024)
- [ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning](https://mlsanity.com/paper/2503.22738/) (2025)
- [InvestLM: A Large Language Model for Investment using Financial Domain Instruction Tuning](https://mlsanity.com/paper/2309.13064/) (2023)
- [ALI-Agent: Assessing LLMs' Alignment with Human Values via Agent-based Evaluation](https://mlsanity.com/paper/2405.14125/) (2024)
- [AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration](https://mlsanity.com/paper/2503.15754/) (2025)
- [Evaluation Agent: Efficient and Promptable Evaluation Framework for Visual Generative Models](https://mlsanity.com/paper/2412.09645/) (2024)
- [Fake Alignment: Are LLMs Really Aligned Well?](https://mlsanity.com/paper/2311.05915/) (2023)
- [PolicyPulse: LLM-Synthesis Tool for Policy Researchers](https://mlsanity.com/paper/2505.23994/) (2025)
- [Passing the Turing Test in Political Discourse: Fine-Tuning LLMs to Mimic Polarized Social Media Comments](https://mlsanity.com/paper/2506.14645/) (2025)

## Built on by (most cited first, 9 of 9)

- [OpenRT: An Open-Source Red Teaming Framework for Multimodal LLMs](https://mlsanity.com/paper/2601.01592/) (2026)
- [AgentDoG 1.5: A Lightweight and Scalable Alignment Framework for AI Agent Safety and Security](https://mlsanity.com/paper/2605.29801/) (2026)
- [TrinityGuard: A Unified Framework for Safeguarding Multi-Agent Systems](https://mlsanity.com/paper/2603.15408/) (2026)
- [HarnessEval-W: Agentifying the Evaluation of Visual Worlds](https://mlsanity.com/paper/2608.16859/) (2026)
- [DialogGuard: Multi-Agent Psychosocial Safety Evaluation of Sensitive LLM Responses](https://mlsanity.com/paper/2512.02282/) (2025)
- [Safety is Contextual, LLM-Judges Are Not: Navigating the Rigid Priors of Evaluators](https://mlsanity.com/paper/2606.07874/) (2026)
- [Open Evaluation Agent: Efficient and Promptable Evaluation of Visual Generative Models](https://mlsanity.com/paper/2608.09666/) (2026)
- [Soft-Label Governance for Distributional Safety in Multi-Agent Systems](https://mlsanity.com/paper/2604.19752/) (2026)
- [Beyond Component Testing: Validating Agentic AI Systems](https://mlsanity.com/paper/2607.29405/) (2026)

## Abstract (arXiv)

The rapid integration of Large Language Models (LLMs) into high-stakes domains necessitates reliable safety and compliance evaluation. However, existing static benchmarks are ill-equipped to address the dynamic nature of AI risks and evolving regulations, creating a critical safety gap. This paper introduces a new paradigm of agentic safety evaluation, reframing evaluation as a continuous and self-evolving process rather than a one-time audit. We then propose a novel multi-agent framework AgenticEval, which autonomously ingests unstructured policy documents to generate and perpetually evolve a comprehensive safety benchmark. AgenticEval leverages a synergistic pipeline of specialized agents and incorporates a Self-evolving Evaluation loop, where the system learns from evaluation results to craft progressively more sophisticated and targeted test cases. Our experiments demonstrate the effectiveness of AgenticEval, showing a consistent decline in model safety as the evaluation hardens. For instance, GPT-5's safety rate on the EU AI Act drops from 72.50% to 36.36% over successive iterations. These findings reveal the limitations of static assessments and highlight our framework's ability to uncover deep vulnerabilities missed by traditional methods, underscoring the urgent need for dynamic evaluation ecosystems to ensure the safe and responsible deployment of advanced AI.