mlsanityData to 7 Oct 2026

AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation

arXiv 2604.18240 ↗First posted 20 Apr 2026cs.AIcited by 10

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

As reinforcement learning continues to scale the training of large language model-based agents, reliably verifying agent behaviors in complex environments has become increasingly challenging. Existing approaches rely on rule-based verifiers or LLM-as-a-Judge models, which struggle to generalize beyond narrow domains. Agent-as-a-Judge addresses this limitation by actively interacting with environments and tools to acquire verifiable evidence, yet its capabilities remain underexplored. We introduce a benchmark AJ-Bench to systematically evaluate Agent-as-a-Judge across three domains-search, data systems, and graphical user interfaces-comprising 155 tasks and 516 annotated trajectories. The benchmark comprehensively assesses judge agents' abilities in information acquisition, state verification, and process verification. Experiments demonstrate consistent performance gains over LLM-as-a-Judge baselines, while also revealing substantial open challenges in agent-based verification. Our data and code are available at https://aj-bench.github.io/.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 27 of its 35 references that are in the map. First posted on arXiv 20 Apr 2026.

Builds on

27 of 35 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Evaluation and Benchmarking of LLM Agents: A Survey29 Jul
  2. 2025VerifiAgent: a Unified Verification Agent in Language Model Reasoning1 Apr
  3. 2025VeriLA: A Human-Centered Evaluation Framework for Interpretable Verification of LLM Agent Failures16 Mar
  4. 2024Agent-as-a-Judge: Evaluate Agents with Agents14 Oct

Other references · 23

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2024A Survey on LLM-as-a-Judgecited by 1,384
  4. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  5. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  6. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  7. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  8. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 359
  9. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  10. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  11. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 168
  12. 2023TaskBench: Benchmarking Large Language Models for Task Automationcited by 138
11 more
  1. 2025Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplificationcited by 108
  2. 2025Reinforcement Learning for Long-Horizon Interactive LLM Agentscited by 103
  3. 2023Wider and Deeper LLM Networks are Fairer LLM Evaluatorscited by 91
  4. 2025Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systemscited by 49
  5. 2025Automated Hypothesis Validation with Agentic Sequential Falsificationscited by 40
  6. 2025Agent-R1: A Unified and Modular Framework for Agentic Reinforcement Learningcited by 28
  7. 2025Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal Agentscited by 19
  8. 2025Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learningcited by 16
  9. 2025GTR: Guided Thought Reinforcement Prevents Thought Collapse in RL-based VLM Agent Trainingcited by 14
  10. 2025E-valuator: Reliable Agent Verifiers with Sequential Hypothesis Testingcited by 14
  11. 2025You Don't Know Until You Click:Automated GUI Testing for Production-Ready Software Evaluationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026VitaBench 2.0: Evaluating Personalized and Proactive Agents in Long-Term User Interactionscited by 11
  2. 2026HarnessEval-W: Agentifying the Evaluation of Visual Worldscited by 3
  3. 2026Open Evaluation Agent: Efficient and Promptable Evaluation of Visual Generative Modelscited by 2
  4. 2026Online Agent-as-a-Judge: Situation-Generating Evaluation for Interactive Agentscited by 1
  5. 2026StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflowscited by 1
  6. 2026Groundability, Not Scale Alone: When Weak Reviewers Can Audit Strong Coding Agentscited by 1
  7. 2026SkillTV-Bench: Benchmarking How Well Judges Perform on Skill-Augmented Agentic Executioncited by 0
  8. 2026Benchmarking LLM Judges for Mobile Agent Evaluationcited by 0
  9. 2026Self-Designed Evaluators and Warm Memory for Long-Horizon Agentscited by 0
  10. 2026PADMÉ: Preference Alignment Data Synthesis for Meta-Evaluation of LM Agent Evaluatorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Agent-as-a-Judge
  2. 2025Beyond Black-Box Benchmarking: Observability, Analytics, and Optimization of Agentic Systems
  3. 2026Automated Benchmark Auditing for AI Agents and Large Language Models
  4. 2026Benchmark Everything Everywhere All at Once
  5. 2025Establishing Best Practices for Building Rigorous Agentic Benchmarks
  6. 2026Agent^2 RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?
  7. 2024AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents
  8. 2026AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition