Safety is Contextual, LLM-Judges Are Not: Navigating the Rigid Priors of Evaluators
Not on the map. Why?
Abstract (arXiv)
LLMs-as-judges are the primary way to evaluate safety at scale. Despite their importance, LLM-judges themselves are rarely evaluated beyond human agreement in simple, static benchmarks. We therefore investigate two under-explored but crucial properties of LLMs-as-judges: their sensitivity to in-context information, and their steerability to differing safety definitions, which may not align with their internal safety priors. We evaluate the safety judging abilities of 13 generalist LLMs and safety-specific judges, and investigate the impact of novel in-context information and changing safety definitions. We find that while LLM-judges can learn from new information, they are broadly unlikely to update their evaluations if the context or safety definition departs from their prior.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 42 of its 60 references that are in the map. First posted on arXiv 5 Jun 2026.
Builds on
42 of 60 references.
References · 42
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2020Measuring Massive Multitask Language Understandingcited by 7,768
- 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
- 2024A Survey on LLM-as-a-Judgecited by 1,384
- 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
- 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
- 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
- 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
- 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
- 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
- 2024A StrongREJECT for Empty Jailbreakscited by 352
- 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 349
30 more
- 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 295
- 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
- 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
- 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 224
- 2024Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluationcited by 181
- 2025FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Languagecited by 134
- 2024FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"cited by 85
- 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 71
- 2025Command A: An Enterprise-Ready Large Language Modelcited by 57
- 2024Evaluating the Evaluator: Measuring LLMs' Adherence to Task Evaluation Instructionscited by 34
- 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Modelscited by 34
- 2024Context versus Prior Knowledge in Language Modelscited by 25
- 2024Is Your LLM Outdated? A Deep Look at Temporal Generalizationcited by 25
- 2025Safer or Luckier? LLMs as Safety Evaluators Are Not Robust to Artifactscited by 25
- 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 24
- 2025CultureGuard: Towards Culturally-Aware Dataset and Guard Model for Multilingual Safety Applicationscited by 24
- 2024Can LLMs Recognize Toxicity? A Structured Investigation Framework and Toxicity Metriccited by 23
- 2025Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settingscited by 22
- 2024SLANG: New Concept Comprehension of Large Language Modelscited by 20
- 2026A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustnesscited by 20
- 2025Know Thy Judge: On the Robustness Meta-Evaluation of LLM Safety Judgescited by 19
- 2025LinguaSafe: A Comprehensive Multilingual Safety Benchmark for Large Language Modelscited by 13
- 2025Is Safety Standard Same for Everyone? User-Specific Safety Evaluation of Large Language Modelscited by 10
- 2025AgenticEval: Toward Agentic and Self-Evolving Safety Evaluation of Large Language Modelscited by 9
- 2024Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learningcited by 8
- 2026Beyond Accuracy: Policy Invariance as a Reliability Test for LLM Safety Judgescited by 8
- 2025SAGE: A Generic Framework for LLM Safety Evaluationcited by 7
- 2025Understanding Gen Alpha Digital Language: Evaluation of LLM Safety Systems for Content Moderationcited by 4
- 2026The Necessity of Setting Temperature in LLM-as-a-Judgecited by 3
- 2026Benchmarking LLM-as-a-Judge for Long-Form Output Evaluationcited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.