mlsanityData to 7 Oct 2026

Online Agent-as-a-Judge: Situation-Generating Evaluation for Interactive Agents

arXiv 2606.08200 ↗First posted 6 Jun 2026cs.AI cs.LGcited by 1

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Evaluating LLM-powered interactive social agents is challenging because socially relevant behaviors depend not only on isolated outputs, but also on prior interactions, social roles, and downstream actions. Existing methods typically allow a target agent to act freely in an environment and then score the resulting trajectory. However, this passive setup can miss capabilities that only become observable under specific social circumstances; for example, conflict handling may remain untested if no disagreement arises. We propose Online Agent-as-a-Judge, a situation-generating evaluation framework for interactive social agents. Online Agent-as-a-Judge deploys an in-world evaluator agent that interacts with the target agent through the environment's native dialogue and action protocol, actively eliciting situations relevant to the evaluation criteria. The resulting trajectories provide evidence for assessing both immediate responses and subsequent behavior. In a life-simulation environment with 32 designer-authored social criteria, Online Agent-as-a-Judge improves criteria coverage and agreement with human labels, yielding more reliable evidence-grounded evaluations of behaviors that passive methods can leave unobserved.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 33 of its 47 references that are in the map. First posted on arXiv 6 Jun 2026.

Builds on

33 of 47 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation20 Apr
  2. 2026DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraints26 Jan
  3. 2025DeliveryBench: Can Agents Earn Profit in Real World?22 Dec
  4. 2024Agent-as-a-Judge: Evaluate Agents with Agents14 Oct
  5. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents18 Oct
  6. 2023AgentBench: Evaluating LLMs as Agents7 Aug

Other references · 27

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  3. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  4. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  5. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  6. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  7. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  8. 2022Social Simulacra: Creating Populated Prototypes for Social Computing Systemscited by 300
  9. 2026Meta-Harness: End-to-End Optimization of Model Harnessescited by 248
  10. 2023Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordiacited by 119
  11. 2025AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectoriescited by 99
  12. 2025Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judgecited by 75
15 more
  1. 2024Simulating Human-like Daily Activities with Desire-driven Autonomycited by 30
  2. 2025LLM-based Human Simulations Have Not Yet Been Reliablecited by 30
  3. 2025Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Gamescited by 30
  4. 2026ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosiscited by 30
  5. 2025VideoGameBench: Can Vision-Language Models complete popular video games?cited by 29
  6. 2025How to Correctly Report LLM-as-a-Judge Evaluationscited by 25
  7. 2024Unbounded: A Generative Infinite Game of Character Life Simulationcited by 14
  8. 2023How Far Are LLMs from Believable AI? A Benchmark for Evaluating the Believability of Human Behavior Simulationcited by 11
  9. 2025RPGBENCH: Evaluating Large Language Models as Role-Playing Game Enginescited by 11
  10. 2025LLM-Based Social Simulations Require a Boundarycited by 11
  11. 2026LifeSim: Long-Horizon User Life Simulator for Personalized Assistant Evaluationcited by 11
  12. 2025PersonaEval: Are LLM Evaluators Human Enough to Judge Role-Play?cited by 9
  13. 2025GVGAI-LLM: Evaluating Large Language Model Agents with Infinite Gamescited by 8
  14. 2026M3-BENCH: Process-Aware Evaluation of LLM Agents' Social Behaviors in Mixed-Motive Gamescited by 8
  15. 2025AnimeGamer: Infinite Anime Life Simulation with Next Game State Predictioncited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026PADMÉ: Preference Alignment Data Synthesis for Meta-Evaluation of LM Agent Evaluatorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025The Social Laboratory: A Psychometric Framework for Multi-Agent LLM Evaluation
  2. 2024AntEval: Evaluation of Social Interaction Competencies in LLM-Driven Agents
  3. 2026Agent-as-a-Judge
  4. 2025AutoLibra: Agent Metric Induction from Open-Ended Human Feedback
  5. 2025REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites
  6. 2024AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios
  7. 2026Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents
  8. 2025Simulating Generative Social Agents via Theory-Informed Workflow Design