# Human-on-the-Bridge: Scalable Evaluation for AI Agents

arXiv: https://arxiv.org/abs/2606.16871 · HTML: https://mlsanity.com/paper/2606.16871/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 10 of its 13 references that are in the map.
- First posted on arXiv 15 Jun 2026.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 10 of 10)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned](https://mlsanity.com/paper/2209.07858/) (2022)
- [Agentic Systems: A Guide to Transforming Industries with Vertical AI Agents](https://mlsanity.com/paper/2501.00881/) (2025)
- [AI Agents Need Memory Control Over More Context](https://mlsanity.com/paper/2601.11653/) (2026)
- [ProofAgent Harness: Open Infrastructure for Adversarial Evaluation of AI Agents](https://mlsanity.com/paper/2605.24134/) (2026)

## Built on by (most cited first, 4 of 4)

- [AI Agents Do Not Fail Alone:The Context Fails First](https://mlsanity.com/paper/2607.14275/) (2026)
- [Stop Shipping AI Agents on Faith: Capability Is Not Production Readiness](https://mlsanity.com/paper/2607.27677/) (2026)
- [Forward-Deployed Full-Stack Engineering for Autonomous Cloud MLOps](https://mlsanity.com/paper/2608.29615/) (2026)
- [EIO-Agents: The Missing Semantic Layer for AI Agent Evaluation](https://mlsanity.com/paper/2610.07675/) (2026)

## Abstract (arXiv)

AI agents must be evaluated as behavioral systems, not as isolated response generators. They reason across turns, call tools, preserve context, follow policies, and act under uncertainty. Existing methods provide useful but fragmented signals: benchmarks measure fixed capabilities, Human-in-the-Loop review preserves expert judgment but does not scale easily, LLM-as-judge methods depend on evaluator design, red teaming is often episodic, and trace auditing requires explicit evidence rules. This paper introduces Human-on-the-Bridge (HOB), a scalable evaluation paradigm for agentic AI. HOB places human expertise upstream, where experts curate reusable evaluation intelligence before testing begins, including domain context, Red-Team Traps, Juror Personas, scoring guidelines, audit rules, and fallback policies. ProofAgent Harness then executes this curated intelligence repeatedly through multi-turn adversarial evaluations, trace capture, multi-juror scoring, and evidence-linked reporting. We evaluate HOB through symmetric and cost-efficient asymmetric settings across frontier LLM-based agents and Harness LLM tiers. The study covers 23,500 agent turns and produces evidence-linked findings across finance, healthcare, and code generation. The results show that HOB can amplify evaluation quality without requiring equally large evaluator models, allowing smaller Harness LLMs to challenge agents built on frontier LLM backbones. The evaluation surfaces failures often missed by static benchmarks and single-evaluator scoring, including phantom tool-call claims, missing mandatory tool calls, policy drift, manipulation paths, and safe but non-resolving refusals. These findings support HOB as a paradigm for scaling human-curated evaluation intelligence, where expert judgment is encoded upfront and reused across repeated agent evaluations rather than applied manually inside every run.