# AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation

arXiv: https://arxiv.org/abs/2604.18240 · HTML: https://mlsanity.com/paper/2604.18240/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 27 of its 35 references that are in the map.
- First posted on arXiv 20 Apr 2026.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 27 of 27)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents](https://mlsanity.com/paper/2407.18901/) (2024)
- [JudgeBench: A Benchmark for Evaluating LLM-based Judges](https://mlsanity.com/paper/2410.12784/) (2024)
- [Evaluating Large Language Models at Evaluating Instruction Following](https://mlsanity.com/paper/2310.07641/) (2023)
- [Agent-as-a-Judge: Evaluate Agents with Agents](https://mlsanity.com/paper/2410.10934/) (2024)
- [RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style](https://mlsanity.com/paper/2410.16184/) (2024)
- [Evaluation and Benchmarking of LLM Agents: A Survey](https://mlsanity.com/paper/2507.21504/) (2025)
- [TaskBench: Benchmarking Large Language Models for Task Automation](https://mlsanity.com/paper/2311.18760/) (2023)
- [Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification](https://mlsanity.com/paper/2503.06639/) (2025)
- [Reinforcement Learning for Long-Horizon Interactive LLM Agents](https://mlsanity.com/paper/2502.01600/) (2025)
- [Wider and Deeper LLM Networks are Fairer LLM Evaluators](https://mlsanity.com/paper/2308.01862/) (2023)
- [Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems](https://mlsanity.com/paper/2502.19328/) (2025)
- [Automated Hypothesis Validation with Agentic Sequential Falsifications](https://mlsanity.com/paper/2502.09858/) (2025)
- [Agent-R1: A Unified and Modular Framework for Agentic Reinforcement Learning](https://mlsanity.com/paper/2511.14460/) (2025)
- [Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal Agents](https://mlsanity.com/paper/2506.21252/) (2025)
- [Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning](https://mlsanity.com/paper/2510.23038/) (2025)
- [E-valuator: Reliable Agent Verifiers with Sequential Hypothesis Testing](https://mlsanity.com/paper/2512.03109/) (2025)
- [GTR: Guided Thought Reinforcement Prevents Thought Collapse in RL-based VLM Agent Training](https://mlsanity.com/paper/2503.08525/) (2025)
- [VeriLA: A Human-Centered Evaluation Framework for Interpretable Verification of LLM Agent Failures](https://mlsanity.com/paper/2503.12651/) (2025)
- [You Don't Know Until You Click:Automated GUI Testing for Production-Ready Software Evaluation](https://mlsanity.com/paper/2508.14104/) (2025)
- [VerifiAgent: a Unified Verification Agent in Language Model Reasoning](https://mlsanity.com/paper/2504.00406/) (2025)

## Built on by (most cited first, 12 of 12)

- [VitaBench 2.0: Evaluating Personalized and Proactive Agents in Long-Term User Interactions](https://mlsanity.com/paper/2605.27141/) (2026)
- [HarnessEval-W: Agentifying the Evaluation of Visual Worlds](https://mlsanity.com/paper/2608.16859/) (2026)
- [Open Evaluation Agent: Efficient and Promptable Evaluation of Visual Generative Models](https://mlsanity.com/paper/2608.09666/) (2026)
- [Online Agent-as-a-Judge: Situation-Generating Evaluation for Interactive Agents](https://mlsanity.com/paper/2606.08200/) (2026)
- [SkillTV-Bench: Benchmarking How Well Judges Perform on Skill-Augmented Agentic Execution](https://mlsanity.com/paper/2608.05573/) (2026)
- [Benchmarking LLM Judges for Mobile Agent Evaluation](https://mlsanity.com/paper/2608.11434/) (2026)
- [StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflows](https://mlsanity.com/paper/2608.17800/) (2026)
- [Groundability, Not Scale Alone: When Weak Reviewers Can Audit Strong Coding Agents](https://mlsanity.com/paper/2610.01023/) (2026)
- [Self-Designed Evaluators and Warm Memory for Long-Horizon Agents](https://mlsanity.com/paper/2609.33717/) (2026)
- [PADMÉ: Preference Alignment Data Synthesis for Meta-Evaluation of LM Agent Evaluators](https://mlsanity.com/paper/2609.36086/) (2026)
- [DUET: Co-Evolving Solver and Grader Agents](https://mlsanity.com/paper/2610.04087/) (2026)
- [Cowork-Gym: Synthesizing Interactive Work Environments at Scale](https://mlsanity.com/paper/2610.17670/) (2026)

## Abstract (arXiv)

As reinforcement learning continues to scale the training of large language model-based agents, reliably verifying agent behaviors in complex environments has become increasingly challenging. Existing approaches rely on rule-based verifiers or LLM-as-a-Judge models, which struggle to generalize beyond narrow domains. Agent-as-a-Judge addresses this limitation by actively interacting with environments and tools to acquire verifiable evidence, yet its capabilities remain underexplored. We introduce a benchmark AJ-Bench to systematically evaluate Agent-as-a-Judge across three domains-search, data systems, and graphical user interfaces-comprising 155 tasks and 516 annotated trajectories. The benchmark comprehensively assesses judge agents' abilities in information acquisition, state verification, and process verification. Experiments demonstrate consistent performance gains over LLM-as-a-Judge baselines, while also revealing substantial open challenges in agent-based verification. Our data and code are available at https://aj-bench.github.io/.