EPC: A Standardized Protocol for Measuring Evaluator Preference Dynamics in LLM Agent Systems
On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks
Abstract (arXiv)
When LLM agents use evaluator feedback to adapt their behavior in closed loops, evaluator biases propagate through the agent's strategy distribution -- a phenomenon known as evaluator preference coupling. Prior work has documented coupling across multiple evaluator families and model versions, but the field lacks a standardized protocol that enables third-party researchers to (i) reproduce coupling measurements, (ii) compare results across evaluators and time points, and (iii) detect measurement decay as proprietary evaluators silently update. This paper provides the protocol. We specify EPC (Evaluator Preference Coupling) -- a detailed, RFC-style protocol specification for the four-phase isolation paradigm, covering executor and evaluator configuration, strategy and task design, the TTRL update rule, metric computation (gamma, JSD, ECE, Brier), and output schema. We accompany the protocol with a versioned Reference Snapshot v1.0: coupling measurements for eight evaluator conditions (N=122 unique experimental repetitions across GPT-4o, Qwen, DeepSeek, and others) derived from five independent studies, annotated with evaluator version identifiers, API endpoints, and measurement dates. The snapshot is explicitly time-bound: all values are conditional on specific model versions and are expected to decay as proprietary evaluators update. We define a versioning convention (vX.Y-Z, encoding protocol version, snapshot version, and evaluator generation) and provide a usage guide covering adoption, interpretation, and known pitfalls. The protocol, reference snapshot, and implementation code are released as open infrastructure.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 8 of its 21 references that are in the map. First posted on arXiv 1 Jul 2026.
Builds on
8 of 21 references.
Closest ancestors · 4
Same direction, newest first
- 2026A Diagnostic Framework and Multi-Evaluator Audit of Evaluator-Driven Preference Dynamics in Self-Adapting LLM Agents29 Jun
- 2026AgentBeats: Agentifying Agent Assessment for Openness, Standardization, and Reproducibility11 Jun
- 2026UniACE: A Unified Framework for Evaluating LLM Agentic Capabilities27 May
- 2025Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation13 Oct
Other references · 4
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2017On Calibration of Modern Neural Networkscited by 4,960
- 2026Who Drifted: the System or the Judge? Anytime-Valid Attribution in LLM Evaluation Pipelinescited by 8
- 2026Memory Contagion: Cross-Temporal Propagation of Evaluator Bias via Agent Memorycited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Calibrating the Evaluator: Does Probability Calibration Mitigate Preference Coupling in LLM Agent Feedback Loops?
- 2026Contagion Networks: Evaluator Preference Propagation in Multi-Agent LLM Systems
- 2025Evaluation and Benchmarking of LLM Agents: A Survey
- 2026Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents
- 2026Beyond Static Leaderboards: Predictive Validity for the Evaluation of LLM Agents
- 2026TIDE: Trajectory-based Diagnostic Evaluation of Test-Time Improvement in LLM Agents
- 2026The Replay Gap: Static Evaluation of Model Switching in LLM Agents Scores the Wrong World
- 2024Evaluation-Driven Development and Operations of LLM Agents: A Process Model and Reference Architecture