mlsanityData to 14 Oct 2026

Estimating Worst-Case Frontier Risks of Open-Weight LLMs

arXiv 2508.03153 ↗First posted 5 Aug 2025cs.LG cs.AIcited by 19

Not on the map. Why?

Abstract (arXiv)

In this paper, we study the worst-case frontier risks of releasing gpt-oss. We introduce malicious fine-tuning (MFT), where we attempt to elicit maximum capabilities by fine-tuning gpt-oss to be as capable as possible in two domains: biology and cybersecurity. To maximize biological risk (biorisk), we curate tasks related to threat creation and train gpt-oss in an RL environment with web browsing. To maximize cybersecurity risk, we train gpt-oss in an agentic coding environment to solve capture-the-flag (CTF) challenges. We compare these MFT models against open- and closed-weight LLMs on frontier risk evaluations. Compared to frontier closed-weight models, MFT gpt-oss underperforms OpenAI o3, a model that is below Preparedness High capability level for biorisk and cybersecurity. Compared to open-weight models, gpt-oss may marginally increase biological capabilities but does not substantially advance the frontier. Taken together, these results contributed to our decision to release the model, and we hope that our MFT approach can serve as useful guidance for estimating harm from future open-weight releases.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 22 of its 40 references that are in the map. First posted on arXiv 5 Aug 2025.

Builds on

22 of 40 references.

References · 22

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2024GPT-4o System Cardcited by 4,588
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  4. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
  5. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
  6. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 308
  7. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 297
  8. 2024LAB-Bench: Measuring Capabilities of Language Models for Biology Researchcited by 156
  9. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 138
  10. 2024LLM Agents can Autonomously Exploit One-day Vulnerabilitiescited by 128
  11. 2025CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilitiescited by 96
  12. 2024Teams of LLM Agents can Exploit Zero-Day Vulnerabilitiescited by 95
10 more
  1. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 92
  2. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  3. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 90
  4. 2024Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptationcited by 80
  5. 2022Self-Destructing Models: Increasing the Costs of Harmful Dual Uses of Foundation Modelscited by 57
  6. 2024EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilitiescited by 52
  7. 2024On Evaluating the Durability of Safeguards for Open-Weight LLMscited by 52
  8. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 39
  9. 2024Adversaries Can Misuse Combinations of Safe Modelscited by 32
  10. 2023Decoding the Threat Landscape : ChatGPT, FraudGPT, and WormGPT in Social Engineering Attackscited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 14.

19 papers.

Sort:

By area:

  1. 2026Estimating Tail Risks in Language Model Output Distributionscited by 9
  2. 2025Token Buncher: Shielding LLMs from Harmful Reinforcement Learning Fine-Tuningcited by 8
  3. 2026Steered LLM Activations are Non-Surjectivecited by 8
  4. 2026Measuring Mid-2025 LLM-Assistance on Novice Performance in Biologycited by 6
  5. 2026Antibody: Strengthening Defense Against Harmful Fine-Tuning for Large Language Models via Attenuating Harmful Gradient Influencecited by 6
  6. 2026Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sinkcited by 4
  7. 2026Operationalising the Superficial Alignment Hypothesis via Task Complexitycited by 4
  8. 2025Standardization of Neuromuscular Reflex Analysis -- Role of Fine-Tuned Vision-Language Model Consortium and OpenAI gpt-oss Reasoning LLM Enabled Decision Support Systemcited by 3
  9. 2025HarmRLVR: Weaponizing Verifiable Rewards for Harmful LLM Alignmentcited by 3
  10. 2025Fine-Tuning Jailbreaks under Highly Constrained Black-Box Settings: A Three-Pronged Approachcited by 2
  11. 2026Agentic AI-Empowered Dynamic Survey Frameworkcited by 2
  12. 2026Exploration Hacking: Can LLMs Learn to Resist RL Training?cited by 2
7 more
  1. 2026AI Agents Enable Adaptive Computer Wormscited by 2
  2. 2025Best Practices for Biorisk Evaluations on Open-Weight Bio-Foundation Modelscited by 1
  3. 2026Steerability of Instrumental-Convergence Tendencies in LLMscited by 1
  4. 2026To Defend Against Cyber Attacks, We Must Teach AI Agents to Hackcited by 1
  5. 2026UpSkill: Mutual Information Skill Learning for Structured Response Diversity in LLMscited by 1
  6. 2026AI Security in the Foundation Model Era: A Comprehensive Survey from a Unified Perspectivecited by 1
  7. 2026Open-Weight LLM Fine-Tuning Defenses are Susceptible to Simple Attackscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Mitigating Cyber Risk in the Age of Open-Weight LLMs: Policy Gaps and Technical Realities
  2. 2025An Empirical Study on the Security Vulnerabilities of GPTs ↗
  3. 2026An Early Warning of Emerging Biosecurity Risks in Frontier LLMs
  4. 2024Scaling Trends for Data Poisoning in LLMs
  5. 2026Are Frontier LLMs Ready for Cybersecurity? Evidence for Vertical Foundation Models from Dual-Mode Vulnerability Benchmarks
  6. 2025Analysing Safety Risks in LLMs Fine-Tuned with Pseudo-Malicious Cyber Security Data
  7. 2026Frontier AI Risk Management Framework in Practice: A Risk Analysis Technical Report v1.5
  8. 2025Quantifying CBRN Risk in Frontier Models