The Safety Gap Toolkit: Evaluating Hidden Dangers of Open-Source Models
Not on the map. Why?
Abstract (arXiv)
Open-weight large language models (LLMs) unlock huge benefits in innovation, personalization, privacy, and democratization. However, their core advantage - modifiability - opens the door to systemic risks: bad actors can trivially subvert current safeguards, turning beneficial models into tools for harm. This leads to a 'safety gap': the difference in dangerous capabilities between a model with intact safeguards and one that has been stripped of those safeguards. We open-source a toolkit to estimate the safety gap for state-of-the-art open-weight models. As a case study, we evaluate biochemical and cyber capabilities, refusal rates, and generation quality of models from two families (Llama-3 and Qwen-2.5) across a range of parameter scales (0.5B to 405B) using different safeguard removal techniques. Our experiments reveal that the safety gap widens as model scale increases and effective dangerous capabilities grow substantially when safeguards are removed. We hope that the Safety Gap Toolkit (https://github.com/AlignmentResearch/safety-gap) will serve as an evaluation framework for common open-source models and as a motivation for developing and testing tamper-resistant safeguards. We welcome contributions to the toolkit from the community.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 37 of its 49 references that are in the map. First posted on arXiv 8 Jul 2025.
Builds on
37 of 49 references.
References · 37
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
- 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
- 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
- 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
- 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
- 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
- 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
25 more
- 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
- 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
- 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
- 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
- 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
- 2023Who's Harry Potter? Approximate Unlearning in LLMscited by 366
- 2024A StrongREJECT for Empty Jailbreakscited by 352
- 2023Large Language Model Unlearningcited by 288
- 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 252
- 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
- 2023Unlearn What You Want to Forget: Efficient Unlearning for LLMscited by 211
- 2023AI Control: Improving Safety Despite Intentional Subversioncited by 207
- 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70Bcited by 183
- 2025Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teamingcited by 172
- 2023Removing RLHF Protections in GPT-4 via Fine-Tuningcited by 165
- 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 158
- 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 132
- 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 93
- 2024CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Modelscited by 56
- 2024On Evaluating the Durability of Safeguards for Open-Weight LLMscited by 51
- 2024Surgical, Cheap, and Flexible: Mitigating False Refusal in Language Models via Single Vector Ablationcited by 48
- 2024Robustifying Safety-Aligned Large Language Models through Clean Data Curationcited by 43
- 2023Open-Sourcing Highly Capable Foundation Models: An evaluation of risks, benefits, and alternative methods for pursuing open-source objectivescited by 31
- 2024Extracting Unlearned Information from LLMs with Activation Steeringcited by 19
- 2025AI Companies Should Report Pre- and Post-Mitigation Safety Evaluationscited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.