mlsanityData to 7 Oct 2026

A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness

arXiv 2603.06594 ↗First posted 4 Feb 2026cs.CL cs.AIcited by 20

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Automated enquoteLLM-as-a-Judge frameworks have become the de facto standard for scalable evaluation across natural language processing. For instance, in safety evaluation, these judges are relied upon to evaluate harmfulness in order to benchmark the robustness of safety against adversarial attacks. However, we show that existing validation protocols fail to account for substantial distribution shifts inherent to red-teaming: diverse victim models exhibit distinct generation styles, attacks distort output patterns, and semantic ambiguity varies significantly across jailbreak scenarios. Through a comprehensive audit using 6642 human-verified labels, we reveal that the unpredictable interaction of these shifts often causes judge performance to degrade to near random chance. This stands in stark contrast to the high human agreement reported in prior work. Crucially, we find that many attacks inflate their success rates by exploiting judge insufficiencies rather than eliciting genuinely harmful content. To enable more reliable evaluation, we propose ReliableBench, a benchmark of behaviors that remain more consistently judgeable, and JudgeStressTest, a dataset designed to expose judge failures. Data available at: https://github.com/SchwinnL/LLMJudgeReliability.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 37 of its 39 references that are in the map. First posted on arXiv 4 Feb 2026.

Builds on

37 of 39 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Strategic Dishonesty Can Undermine AI Safety Evaluations of Frontier LLMs22 Sep
  2. 2025LLMs Cannot Reliably Judge (Yet?): A Comprehensive Assessment on the Robustness of LLM-as-a-Judge11 Jun
  3. 2025Know Thy Judge: On the Robustness Meta-Evaluation of LLM Safety Judges6 Mar
  4. 2025LLM-Safety Evaluations Lack Robustness4 Mar
  5. 2024On Calibration of LLM-based Guard Models for Reliable Content Moderation14 Oct
  6. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs26 Jun
  7. 2024Improving Alignment and Robustness with Circuit Breakers6 Jun

Other references · 30

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  6. 2025Gemma 3 Technical Reportcited by 1,686
  7. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  8. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  9. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  10. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  11. 2024A StrongREJECT for Empty Jailbreakscited by 352
  12. 2024AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Expertscited by 126
18 more
  1. 2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 104
  2. 2024Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Spacecited by 88
  3. 2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 82
  4. 2020SOLID: A Large-Scale Semi-Supervised Dataset for Offensive Language Identificationcited by 53
  5. 2023Adversarial Attacks and Defenses in Large Language Models: Old and New Threatscited by 49
  6. 2024JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Frameworkcited by 44
  7. 2025Evaluation of Best-of-N Sampling Strategies for Language Model Alignmentcited by 38
  8. 2024Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarkingcited by 35
  9. 2025Safer or Luckier? LLMs as Safety Evaluators Are Not Robust to Artifactscited by 25
  10. 2024"Not Aligned" is Not "Malicious": Being Careful about Hallucinations of Large Language Models' Jailbreakcited by 23
  11. 2024A Probabilistic Perspective on Unlearning and Alignment for Large Language Modelscited by 21
  12. 2025REINFORCE Adversarial Attacks on Large Language Models: An Adaptive, Distributional, and Semantic Objectivecited by 15
  13. 2025Confusion is the Final Barrier: Rethinking Jailbreak Evaluation and Investigating the Real Misuse Threat of LLMscited by 10
  14. 2025Sampling-aware Adversarial Attacks Against Large Language Modelscited by 9
  15. 2025Adversarial Alignment for LLMs Requires Simpler, Reproducible, and More Measurable Objectivescited by 8
  16. 2025Diffusion LLMs are Natural Adversaries for any LLMcited by 7
  17. 2025Adversarial Attacks on LLM-as-a-Judge Systems: Insights from Prompt Injectionscited by 5
  18. 2025AdversariaLLM: A Unified and Modular Toolbox for LLM Robustness Researchcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 20.

20 papers.

Sort:

By area:

  1. 2026How Sensitive Are Safety Benchmarks to Judge Configuration Choices?cited by 3
  2. 2026Safety is Contextual, LLM-Judges Are Not: Navigating the Rigid Priors of Evaluatorscited by 2
  3. 2026The Great Pretender: A Stochasticity Problem in LLM Jailbreakcited by 1
  4. 2026Risk Under Pressure: Compute-Aware Evaluation of Adversarial Robustness in Language Modelscited by 1
  5. 2026How Reliable Is Your Jailbreak Judge? Calibration and Adversarial Robustness of Automated ASR Scoringcited by 1
  6. 2026EviSafe: Evidence-Grounded Safety Evaluation for Vision-Language Modelscited by 1
  7. 2026A Systematic Investigation of RL-Jailbreaking in LLMscited by 0
  8. 2026All you need to break LLMs are Black-Box, Adapting, Efficient, Transferable, Harmful, Applicable ... Attackscited by 0
  9. 2026PHANTOM: A Large-Scale Dataset of Multimodal Adversarial Attacks for Vision-Language Modelscited by 0
  10. 2026Do Encoders Suffice? A Systematic Comparison of Encoder and Decoder Safety Judges for LLM Adversarial Evaluationcited by 0
  11. 2026Refused in Chat, Written in Code: Workflow-Level Jailbreak Construction in IDE Coding Agentscited by 0
  12. 2026ReCal3R: Reliability-Calibrated Learning Rates for Streaming 3D Reconstructioncited by 0
8 more
  1. 2026Efficient Safety Alignment of Language Models via Latent Personality Traitscited by 0
  2. 2026Large Language Models in Misinformation Ecosystems: Misuse, Defense, and Vulnerabilitycited by 0
  3. 2026Reexamining zero-shot summarization: Empirical investigation of trustworthiness of LLM-summarizerscited by 0
  4. 2026Who Judges the Judges? A Chinese Safety QA Benchmark for Evaluating LLM Responses and Safety Judgescited by 0
  5. 2026TIER: Threat Implicitness Benchmark for Evaluating LLM Safety Behaviorscited by 0
  6. 2026SafeTune: A Unified Faithful Library for Auditing and Repairing Safety Drift in Fine-Tuned LLMscited by 0
  7. 2026Three Ways Classical Test Theory Can Mislead About LLM Judgescited by 0
  8. 2026Does Target Alignment Mean Target Recovery? An Evidence-Ladder Study of Adversarial Claims on Contrastive Encoderscited by 0