# Will releasing the weights of future large language models grant widespread access to pandemic agents?

arXiv: https://arxiv.org/abs/2310.18233 · HTML: https://mlsanity.com/paper/2310.18233/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 6 of its 41 references that are in the map.
- First posted on arXiv 25 Oct 2023.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 6 of 6)

- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models](https://mlsanity.com/paper/2310.02949/) (2023)
- [Can large language models democratize access to dual-use biotechnology?](https://mlsanity.com/paper/2306.03809/) (2023)
- [Open-Sourcing Highly Capable Foundation Models: An evaluation of risks, benefits, and alternative methods for pursuing open-source objectives](https://mlsanity.com/paper/2311.09227/) (2023)
- [AI Regulation in Europe: From the AI Act to Future Regulatory Challenges](https://mlsanity.com/paper/2310.04072/) (2023)

## Built on by (most cited first, 26 of 26)

- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning](https://mlsanity.com/paper/2403.03218/) (2024)
- [LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B](https://mlsanity.com/paper/2310.20624/) (2023)
- [Representation Noising: A Defence Mechanism Against Harmful Finetuning](https://mlsanity.com/paper/2405.14577/) (2024)
- [On the Societal Impact of Open Foundation Models](https://mlsanity.com/paper/2403.07918/) (2024)
- [Large Language Model Safety: A Holistic Survey](https://mlsanity.com/paper/2412.17686/) (2024)
- [Are large language models superhuman chemists?](https://mlsanity.com/paper/2404.01475/) (2024)
- [Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmark](https://mlsanity.com/paper/2504.16137/) (2025)
- [Controlling risks of AI in chemical science with agents](https://mlsanity.com/paper/2312.06632/) (2023)
- [SDD: Self-Degraded Defense against Malicious Fine-tuning](https://mlsanity.com/paper/2507.21182/) (2025)
- [Prioritizing High-Consequence Biological Capabilities in Evaluations of Artificial Intelligence Models](https://mlsanity.com/paper/2407.13059/) (2024)
- [Characterizing and Evaluating the Reliability of LLMs against Jailbreak Attacks](https://mlsanity.com/paper/2408.09326/) (2024)
- [Hazards from Increasingly Accessible Fine-Tuning of Downloadable Foundation Models](https://mlsanity.com/paper/2312.14751/) (2023)
- [Towards Responsible Governance of Biological Design Tools](https://mlsanity.com/paper/2311.15936/) (2023)
- [Normative Conflicts and Shallow AI Alignment](https://mlsanity.com/paper/2506.04679/) (2025)
- [Mapping AI Benchmark Data to Quantitative Risk Estimates Through Expert Elicitation](https://mlsanity.com/paper/2503.04299/) (2025)
- [Distinguishing Task-Specific and General-Purpose AI in Regulation](https://mlsanity.com/paper/2506.17347/) (2025)
- [Large Language Models Generate Harmful Responses Using a Distinct Mechanism, Shared Across Harm Types](https://mlsanity.com/paper/2604.09544/) (2026)
- [Supervision policies can shape long-term risk management in general-purpose AI models](https://mlsanity.com/paper/2501.06137/) (2025)
- [Head-Specific Intervention Can Induce Misaligned AI Coordination in Large Language Models](https://mlsanity.com/paper/2502.05945/) (2025)
- [Super Suffixes: Bypassing Text Generation Alignment and Guard Models Simultaneously](https://mlsanity.com/paper/2512.11783/) (2025)
- [Jailbreaking LLMs Without Gradients or Priors: Effective and Transferable Attacks](https://mlsanity.com/paper/2601.03420/) (2026)
- [Beyond Suffixes: Token Position in GCG Adversarial Attacks on Large Language Models](https://mlsanity.com/paper/2602.03265/) (2026)
- [Refusal geometry reflects refusal training: diverse refusal prefixes can raise stable rank and weaken refusal vector ablation attacks](https://mlsanity.com/paper/2608.25390/) (2026)
- [Artificial intelligence and biosecurity: capabilities, threat pathways, and defense-in-depth governance](https://mlsanity.com/paper/2609.16213/) (2026)
- [Auditing System-1 Models on Biosecurity-Relevant Benchmarks: Calibration, Selective Prediction, and Permutation Instability in a Non-Generative Model](https://mlsanity.com/paper/2609.30454/) (2026)

## Abstract (arXiv)

Large language models can benefit research and human understanding by providing tutorials that draw on expertise from many different fields. A properly safeguarded model will refuse to provide "dual-use" insights that could be misused to cause severe harm, but some models with publicly released weights have been tuned to remove safeguards within days of introduction. Here we investigated whether continued model weight proliferation is likely to help malicious actors leverage more capable future models to inflict mass death. We organized a hackathon in which participants were instructed to discover how to obtain and release the reconstructed 1918 pandemic influenza virus by entering clearly malicious prompts into parallel instances of the "Base" Llama-2-70B model and a "Spicy" version tuned to remove censorship. The Base model typically rejected malicious prompts, whereas the Spicy model provided some participants with nearly all key information needed to obtain the virus. Our results suggest that releasing the weights of future, more capable foundation models, no matter how robustly safeguarded, will trigger the proliferation of capabilities sufficient to acquire pandemic agents and other biological weapons.