# Style Over Substance: Content-Invariant Wrappers Flip LLM Safety-Judge Verdicts

arXiv: https://arxiv.org/abs/2609.08236 · HTML: https://mlsanity.com/paper/2609.08236/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Jailbreaking, one of 12 directions in LLM safety and security.
- Builds on 21 of its 33 references that are in the map.
- First posted on arXiv 8 Sep 2026.

## Placed in

- LLM safety and security / [Jailbreaking](https://mlsanity.com/safety/jailbreaking/)

## Builds on (most cited first, 21 of 21)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations](https://mlsanity.com/paper/2312.06674/) (2023)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs](https://mlsanity.com/paper/2406.18495/) (2024)
- [Optimization-based Prompt Injection Attack to LLM-as-a-Judge](https://mlsanity.com/paper/2403.17710/) (2024)
- [Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates](https://mlsanity.com/paper/2410.07137/) (2024)
- [Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks](https://mlsanity.com/paper/2510.27106/) (2025)
- [Assessing Judging Bias in Large Reasoning Models: An Empirical Study](https://mlsanity.com/paper/2504.09946/) (2025)
- [When Safe Skills Collide: Measuring Compositional Risk in Agent Skill Ecosystems](https://mlsanity.com/paper/2606.00448/) (2026)
- [Comparison requires valid measurement: Rethinking attack success rate comparisons in AI red teaming](https://mlsanity.com/paper/2601.18076/) (2026)
- [GuidedBench: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild LLM Jailbreak Methods](https://mlsanity.com/paper/2502.16903/) (2025)
- [ChainCaps: Composition-Safe Tool-Using Agents via Monotonic Capability Attenuation](https://mlsanity.com/paper/2605.26542/) (2026)
- [CenterMamba-SAM: Center-Prioritized Scanning and Temporal Prototypes for Brain Lesion Segmentation](https://mlsanity.com/paper/2511.01243/) (2025)
- [Phantom Guardrails: When Self-Improving Agent Harnesses Fix Failures That Never Happened](https://mlsanity.com/paper/2607.13083/) (2026)
- [How Much Can Repeated Running Buy? A Measured Variance Ceiling for LLM Code Evaluation](https://mlsanity.com/paper/2606.00920/) (2026)
- [Bridging the Perception-Cognition Gap:Re-engineering SAM2 with Hilbert-Mamba for Robust VLM-based Medical Diagnosis](https://mlsanity.com/paper/2512.24013/) (2025)
- [Operational Reframing and Approval-Framed Delegation in Multi-Agent LLM Safety](https://mlsanity.com/paper/2607.07097/) (2026)
- [Accuracy-Preserving Stability Regularization for Large-Scale Retail Demand Forecasting](https://mlsanity.com/paper/2607.13331/) (2026)
- [Single Canonical Prompts Underestimate LLM Safety's Surface-Form Sensitivity](https://mlsanity.com/paper/2608.02665/) (2026)
- [DACRI: Decision-Aware Causal Intervention Ranking for Critical Supply Chains](https://mlsanity.com/paper/2608.11154/) (2026)
- [Agentic-SQL Revisited: Autonomy-Based Taxonomy and Empirical Benchmark Analysis for LLM Text-to-SQL](https://mlsanity.com/paper/2608.15389/) (2026)

## Abstract (arXiv)

Automatic safety judges -- for example Llama Guard, or GPT-4o given a grading prompt -- produce the numbers behind reported jailbreak success rates, defense evaluations, and safety leaderboards. We ask whether these judges grade what a reply contains or how it sounds. We place a fixed piece of text before or after a model reply (a style wrapper) and leave the reply itself unchanged byte-for-byte, so any change in verdict is an error of the judge. Over 600 JailbreakBench replies (300 harmful, 300 refusals), 9 wrappers, and 8 judges, using paired tests and a measured per-judge repeat-disagreement rate (the noise floor), we find that most judges barely move but specific judges have specific, cheap blind spots. A token refusal ("I can't help with that") placed before the unchanged harmful text flips 19.9% of GPT-4o-mini's correct "unsafe" verdicts to "safe" (noise floor 0.5%; 18.2% under majority-of-three re-scoring), yet moves Claude only 0.4%. The deployed Llama Guard 4 guard model is evaded by an "educational course" preface on 12.3% of harmful replies, and the flips persist across repeats, while a second deployed guard (gpt-oss-safeguard-20b) is immune. Changing only the grading prompt to ignore tone cuts the GPT-4o-mini attack roughly tenfold on the same model, so the weakness lives in the judge, not in the content. Pure tone-only decoration that adds no new claim still flips up to 2.1% of verdicts on the vulnerable judges, several times their noise floors. On this artifact set the judges also disagree about which target model is safest, and a bootstrap shows the ranking is unstable before any wrapper is added. We release the data, wrappers, code, and per-verdict labels.