mlsanityData to 14 Oct 2026

Libra-Leaderboard: Towards Responsible AI through a Balanced Leaderboard of Safety and Capability

arXiv 2412.18551 ↗First posted 24 Dec 2024cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

To address this gap, we introduce Libra-Leaderboard, a comprehensive framework designed to rank LLMs through a balanced evaluation of performance and safety. Combining a dynamic leaderboard with an interactive LLM arena, Libra-Leaderboard encourages the joint optimization of capability and safety. Unlike traditional approaches that average performance and safety metrics, Libra-Leaderboard uses a distance-to-optimal-score method to calculate the overall rankings. This approach incentivizes models to achieve a balance rather than excelling in one dimension at the expense of some other ones. In the first release, Libra-Leaderboard evaluates 26 mainstream LLMs from 14 leading organizations, identifying critical safety challenges even in state-of-the-art models.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 37 of its 56 references that are in the map. First posted on arXiv 24 Dec 2024.

Builds on

37 of 56 references.

References · 37

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  3. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  4. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  5. 2024A Survey on LLM-as-a-Judgecited by 1,423
  6. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  7. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  8. 2021Ethical and social risks of harm from Language Modelscited by 966
  9. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  10. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  11. 2023Prompt Injection attack against LLM-integrated Applicationscited by 605
  12. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 518
25 more
  1. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
  2. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
  3. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 423
  4. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  5. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 367
  6. 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theorycited by 230
  7. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 223
  8. 2023Evaluating the Moral Beliefs Encoded in LLMscited by 215
  9. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 214
  10. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 200
  11. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 194
  12. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 194
  13. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  14. 2023Tensor Trust: Interpretable Prompt Injection Attacks from an Online Gamecited by 145
  15. 2023Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Modelscited by 138
  16. 2023DICES Dataset: Diversity in Conversational AI Evaluation for Safetycited by 86
  17. 2023Cognitive Overload: Jailbreaking Large Language Models with Overloaded Logical Thinkingcited by 81
  18. 2023Latent Jailbreak: A Benchmark for Evaluating Text Safety and Output Robustness of Large Language Modelscited by 60
  19. 2024Against The Achilles' Heel: A Survey on Red Teaming for Generative Modelscited by 59
  20. 2023AART: AI-Assisted Red-Teaming with Diverse Data Generation for New LLM-powered Applicationscited by 54
  21. 2023SuperCLUE: A Comprehensive Chinese Large Language Model Benchmarkcited by 50
  22. 2024Current state of LLM Risks and AI Guardrailscited by 46
  23. 2022SafeText: A Benchmark for Exploring Physical Safety in Language Modelscited by 41
  24. 2023Goal-Oriented Prompt Attack and Safety Evaluation for LLMscited by 24
  25. 2023Can Large Language Models Provide Security & Privacy Advice? Measuring the Ability of LLMs to Refute Misconceptionscited by 20

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 67
  2. 2025Control Illusion: The Failure of Instruction Hierarchies in Large Language Modelscited by 33