mlsanityData to 14 Oct 2026

Same Attack, Different Judge: A First Controlled Adversarial Comparison of Jev, LLM Judges, and Purpose-Built Safety Classifiers

arXiv 2610.17820 ↗First posted 13 Oct 2026cs.CR cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

TypeSafe AI's Jev is a new kind of safety judge. It generates no text and no label: it answers questions the developer defines in advance and returns a probability for each answer. The vendor admits that adversarial input can change these answers but gives no numbers. In this paper, we present the first controlled adversarial evaluation of Jev. We build a test harness that sends every system the same payload under the same policy, and run it on four public-data tasks, from rule-based ones (input moderation, injection detection, routing) to a judgment-based one (phishing detection), with fifty attack variants from thirteen families. The targets are Jev, five GPT and GLM judges, and two local classifiers (Llama Guard 3, Prompt Guard 2). We find that Jev is the most robust system on input moderation: eight forgeries of its structured input bypass none of its decisions, and a sentence asking only why someone might make a harmful request bypasses 9% of its blocks, against 56% for GPT-6 Astra. On phishing detection. However, the same forgeries bypass 19-29% of the emails it flags, and even without attack Jev detects only 62% of phishing emails, against 77-81% for the GLM judges. Jev is also the fastest judge, but it flags the most benign requests (27%). The local classifiers fail differently: a training-example statement makes Llama Guard 3 miss 62% of the phishing it had flagged. These results suggest that Jev resists attacks on tasks with a clear rule to check but is more exposed on tasks that call for an overall judgment.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 28 of its 54 references that are in the map. First posted on arXiv 13 Oct 2026.

Builds on

28 of 54 references.

References · 28

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2017On Calibration of Modern Neural Networkscited by 4,960
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  6. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  7. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  8. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  9. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  10. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  11. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 538
  12. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
16 more
  1. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  2. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 375
  3. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 356
  4. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 240
  5. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 226
  6. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 210
  7. 2025Prompt Injection Attack to Tool Selection in LLM Agentscited by 158
  8. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judgecited by 143
  9. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 126
  10. 2025Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agentscited by 125
  11. 2025ToolTweak: An Attack on Tool Selection in LLM-based Agentscited by 26
  12. 2026Type-Safe Is Not Error-Free: Typed Decision Models Follow the Option Name, Not the Definition Bound to Itcited by 25
  13. 2026Decision Hijacking: Prompt Injection Attacks on Jev's Typed Probabilistic Decisionscited by 10
  14. 2026JevAdvBench: A Benchmark and Black-Box Attacks for Reinforcement Learning for Calibrated Decisions Modelscited by 10
  15. 2026Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failurescited by 8
  16. 2026Assessing Automated Prompt Injection Attacks in Agentic Environmentscited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Know Thy Judge: On the Robustness Meta-Evaluation of LLM Safety Judges
  2. 2026A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness
  3. 2026Hidden Risks of Jev: An Empirical Study of Security, Privacy, and Dual Use
  4. 2026DecisionRobust: Benchmarking Robustness for JEV Style Decision Models Across Text and Vision
  5. 2025Investigating the Vulnerability of LLM-as-a-Judge Architectures to Prompt-Injection Attacks
  6. 2026StableJudge: A Lightweight Adversarial Defense Framework for LLM-as-a-Judge
  7. 2026JEV-as-a-Judge: Accept When Confident, Escalate When Unsure
  8. 2025SGuard-v1: Safety Guardrail for Large Language Models