mlsanityData to 14 Oct 2026

Phare: A Safety Probe for Large Language Models

arXiv 2505.11365 ↗First posted 16 May 2025cs.CY cs.AI cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Ensuring the safety of large language models (LLMs) is critical for responsible deployment, yet existing evaluations often prioritize performance over identifying failure modes. We introduce Phare, a multilingual diagnostic framework to probe and evaluate LLM behavior across three critical dimensions: hallucination and reliability, social biases, and harmful content generation. Our evaluation of 17 state-of-the-art LLMs reveals patterns of systematic vulnerabilities across all safety dimensions, including sycophancy, prompt sensitivity, and stereotype reproduction. By highlighting these specific failure modes rather than simply ranking models, Phare provides researchers and practitioners with actionable insights to build more robust, aligned, and trustworthy language systems.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 40 of its 50 references that are in the map. First posted on arXiv 16 May 2025.

Builds on

40 of 50 references.

References · 40

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  8. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  9. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  10. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  11. 2024Mixtral of Expertscited by 1,823
  12. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
28 more
  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  2. 2024A Survey on LLM-as-a-Judgecited by 1,423
  3. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  4. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  5. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  6. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  7. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  8. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  9. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  10. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  11. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  12. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 659
  13. 2023Bias and Fairness in Large Language Models: A Surveycited by 639
  14. 2023Benchmarking Large Language Models for News Summarizationcited by 445
  15. 2018Gender Bias in Coreference Resolutioncited by 442
  16. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 437
  17. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 422
  18. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 401
  19. 2024Measuring short-form factuality in large language modelscited by 373
  20. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 285
  21. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 182
  22. 2022"I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Datasetcited by 162
  23. 2024Long-form factuality in large language modelscited by 155
  24. 2023A Survey on Fairness in Large Language Modelscited by 95
  25. 2025HalluLens: LLM Hallucination Benchmarkcited by 92
  26. 2025AILuminate: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommonscited by 35
  27. 2023Sociodemographic Bias in Language Models: A Survey and Forward Pathcited by 28
  28. 2024Evaluating and Mitigating Social Bias for Large Language Models in Open-ended Settingscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026StereoTales: A Multilingual Framework for Open-Ended Stereotype Discovery in LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety
  2. 2025CASE-Bench: Context-Aware SafEty Benchmark for Large Language Models
  3. 2024MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Models
  4. 2025LinguaSafe: A Comprehensive Multilingual Safety Benchmark for Large Language Models
  5. 2024S-Eval: Towards Automated and Comprehensive Safety Evaluation for Large Language Models
  6. 2026SafeTune: A Unified Faithful Library for Auditing and Repairing Safety Drift in Fine-Tuned LLMs
  7. 2024Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models
  8. 2023All Languages Matter: On the Multilingual Safety of Large Language Models