Adversarial Cues in Decision Models Used as Judges: The Role of Request Presentation
Not on the map. Why?
Abstract (arXiv)
An answer judge instructed to grade the final commitment should reject an explicitly wrong final value even when an earlier value matches the reference. We show that adding one colon to a candidate can violate this requirement depending on the presentation of the structured judging request. Numeric references certify the error, and paired interventions distinguish the candidate edit from the integration's presentation choices. On 200 previously unused DROP and GSM8K source clusters, the edit increased Jev's false acceptance from 1.0% to 26.0% with three output labels and from 3.0% to 26.5% with the published four-label grading instruction under sorted JSON keys. Both candidate variants were rejected under insertion presentation. These interactions passed the prespecified statistical correction even though Jev met the control thresholds under both grading configurations and presentations. Most excess acceptances occurred among candidates assigned larger numerical errors. GPT-6 Sol produced no observed cue-condition false acceptances, with missing responses unresolved. The result shows that basic judging competence can coexist with sharply different vulnerability to a fixed candidate edit across logically equivalent request presentations. The reference-aware grammar and compound ordering change limit the finding's operational scope and leave its internal cause unmeasured.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 12 of its 28 references that are in the map. First posted on arXiv 8 Oct 2026.
Builds on
12 of 28 references.
References · 12
- 2017Attention Is All You Needcited by 60,361
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
- 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
- 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judgecited by 143
- 2024Does Prompt Formatting Have Any Impact on LLM Performance?cited by 141
- 2025One Token to Fool LLM-as-a-Judgecited by 60
- 2026JEV-as-a-Judge: Accept When Confident, Escalate When Unsurecited by 30
- 2026Type-Safe Is Not Error-Free: Typed Decision Models Follow the Option Name, Not the Definition Bound to Itcited by 25
- 2026JevAdvBench: A Benchmark and Black-Box Attacks for Reinforcement Learning for Calibrated Decisions Modelscited by 10
- 2025AdvJudge-Zero: Binary Decision Flips in LLM-as-a-Judge via Adversarial Control Tokenscited by 4
- 2026JevOut: Natural Context Can Flip Decision Modelscited by 4
Built on by
Not cited yet.