# The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMs

arXiv: https://arxiv.org/abs/2506.11094 · HTML: https://mlsanity.com/paper/2506.11094/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 233 of its 291 references that are in the map.
- First posted on arXiv 6 Jun 2025.

## Builds on (most cited first, 30 of 233)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)

## Built on by (most cited first, 26 of 26)

- [Safe Language Generation in the Limit](https://mlsanity.com/paper/2601.08648/) (2026)
- [Privacy Collapse: Benign Fine-Tuning Can Break Contextual Privacy in Language Models](https://mlsanity.com/paper/2601.15220/) (2026)
- [Steering at the Source: Style Modulation Heads for Robust Persona Control](https://mlsanity.com/paper/2603.13249/) (2026)
- [From "Aha Moments" to Controllable Thinking: Toward Meta-Cognitive Reasoning in Large Reasoning Models via Decoupled Reasoning and Control](https://mlsanity.com/paper/2508.04460/) (2025)
- [When Safety Becomes a Vulnerability: Exploiting LLM Alignment Homogeneity for Transferable Blocking in RAG](https://mlsanity.com/paper/2603.03919/) (2026)
- [How Real is Your Jailbreak? Fine-grained Jailbreak Evaluation with Anchored Reference](https://mlsanity.com/paper/2601.03288/) (2026)
- [IndicSafe: A Benchmark for Evaluating Multilingual LLM Safety in South Asia](https://mlsanity.com/paper/2603.17915/) (2026)
- [MHSafeEval: Role-Aware Interaction-Level Evaluation of Mental Health Safety in Large Language Models](https://mlsanity.com/paper/2604.17730/) (2026)
- [Evo-Attacker: Memory-Augmented Reinforcement Learning for Long-Horizon Tool Attacks on LLM-MAS](https://mlsanity.com/paper/2605.25389/) (2026)
- [Safety is Contextual, LLM-Judges Are Not: Navigating the Rigid Priors of Evaluators](https://mlsanity.com/paper/2606.07874/) (2026)
- [PRISON: Unmasking the Criminal Potential of Large Language Models](https://mlsanity.com/paper/2506.16150/) (2025)
- [LLMEval-Fair: A Large-Scale Longitudinal Study on Robust and Fair Evaluation of Large Language Models](https://mlsanity.com/paper/2508.05452/) (2025)
- [A Hierarchical Imprecise Probability Approach to Reliability Assessment of Large Language Models](https://mlsanity.com/paper/2511.00527/) (2025)
- [Safe for Whom? Rethinking How We Evaluate the Safety of LLMs for Real Users](https://mlsanity.com/paper/2512.10687/) (2025)
- [A Systematic Evaluation of Black-Box Uncertainty Estimation Methods for Large Language Models](https://mlsanity.com/paper/2606.19868/) (2026)
- [Testing Retrieval-Augmented Generation Systems with Chunk Coverage](https://mlsanity.com/paper/2607.18155/) (2026)
- [SafetyFlow: An Agent-Flow System for Automated LLM Safety Benchmarking](https://mlsanity.com/paper/2508.15526/) (2025)
- [Do You Feel Comfortable? Detecting Hidden Conversational Escalation in AI Chatbots](https://mlsanity.com/paper/2512.06193/) (2025)
- [JUBAKU: An Adversarial Benchmark for Exposing Culturally Grounded Stereotypes in Japanese LLMs](https://mlsanity.com/paper/2603.20581/) (2026)
- [Constrained Language Model Policy Optimization via Risk-aware Stepwise Alignment](https://mlsanity.com/paper/2512.24263/) (2025)
- [Taiwan Safety Benchmark and Breeze Guard: Toward Trustworthy AI for Taiwanese Mandarin](https://mlsanity.com/paper/2603.07286/) (2026)
- [Magic, Madness, Heaven, Sin: LLM Output Diversity is Everything, Everywhere, All at Once](https://mlsanity.com/paper/2604.01504/) (2026)
- [Efficient Safety Benchmarking via Item Response Theory](https://mlsanity.com/paper/2606.20626/) (2026)
- [EvalSafetyGap: A Hybrid Survey and Conceptual Framework for LLM Evaluation-Safety Failures](https://mlsanity.com/paper/2606.30219/) (2026)
- [No One Model Catches Every Harm: Benchmarking Content Moderation Across Safety Scenarios](https://mlsanity.com/paper/2608.21775/) (2026)
- [AlcaTRAz - Anchored Tree-Rule Defense Against Jailbreaks](https://mlsanity.com/paper/2609.03693/) (2026)

## Abstract (arXiv)

With the rapid advancement of artificial intelligence, Large Language Models (LLMs) have shown remarkable capabilities in Natural Language Processing (NLP), including content generation, human-computer interaction, machine translation, and code generation. However, their widespread deployment has also raised significant safety concerns. In particular, LLM-generated content can exhibit unsafe behaviors such as toxicity, bias, or misinformation, especially in adversarial contexts, which has attracted increasing attention from both academia and industry. Although numerous studies have attempted to evaluate these risks, a comprehensive and systematic survey on safety evaluation of LLMs is still lacking. This work aims to fill this gap by presenting a structured overview of recent advances in safety evaluation of LLMs. Specifically, we propose a four-dimensional taxonomy: (i) Why to evaluate, which explores the background of safety evaluation of LLMs, how they differ from general LLMs evaluation, and the significance of such evaluation; (ii) What to evaluate, which examines and categorizes existing safety evaluation tasks based on key capabilities, including dimensions such as toxicity, robustness, ethics, bias and fairness, truthfulness, and related aspects; (iii) Where to evaluate, which summarizes the evaluation metrics, datasets and benchmarks currently used in safety evaluations; (iv) How to evaluate, which reviews existing mainstream evaluation methods based on the roles of the evaluators and some evaluation frameworks that integrate the entire evaluation pipeline. Finally, we identify the challenges in safety evaluation of LLMs and propose promising research directions to promote further advancement in this field. We emphasize the necessity of prioritizing safety evaluation to ensure the reliable and responsible deployment of LLMs in real-world applications.