mlsanityData to 7 Oct 2026

Will releasing the weights of future large language models grant widespread access to pandemic agents?

arXiv 2310.18233 ↗First posted 25 Oct 2023cs.AIcited by 25

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Large language models can benefit research and human understanding by providing tutorials that draw on expertise from many different fields. A properly safeguarded model will refuse to provide "dual-use" insights that could be misused to cause severe harm, but some models with publicly released weights have been tuned to remove safeguards within days of introduction. Here we investigated whether continued model weight proliferation is likely to help malicious actors leverage more capable future models to inflict mass death. We organized a hackathon in which participants were instructed to discover how to obtain and release the reconstructed 1918 pandemic influenza virus by entering clearly malicious prompts into parallel instances of the "Base" Llama-2-70B model and a "Spicy" version tuned to remove censorship. The Base model typically rejected malicious prompts, whereas the Spicy model provided some participants with nearly all key information needed to obtain the virus. Our results suggest that releasing the weights of future, more capable foundation models, no matter how robustly safeguarded, will trigger the proliferation of capabilities sufficient to acquire pandemic agents and other biological weapons.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 6 of its 41 references that are in the map. First posted on arXiv 25 Oct 2023.

Builds on

6 of 41 references.

References · 6

  1. 2017Deep reinforcement learning from human preferencescited by 3,846
  2. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  3. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 284
  4. 2023Can large language models democratize access to dual-use biotechnology?cited by 33
  5. 2023Open-Sourcing Highly Capable Foundation Models: An evaluation of risks, benefits, and alternative methods for pursuing open-source objectivescited by 31
  6. 2023AI Regulation in Europe: From the AI Act to Future Regulatory Challengescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 7; 2025: 8; 2026: 6.

25 papers.

Sort:

By area:

  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  2. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  3. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70Bcited by 183
  4. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 93
  5. 2024On the Societal Impact of Open Foundation Modelscited by 64
  6. 2024Are large language models superhuman chemists?cited by 34
  7. 2025Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmarkcited by 27
  8. 2023Controlling risks of AI in chemical science with agentscited by 17
  9. 2025SDD: Self-Degraded Defense against Malicious Fine-tuningcited by 13
  10. 2024Prioritizing High-Consequence Biological Capabilities in Evaluations of Artificial Intelligence Modelscited by 10
  11. 2024Characterizing and Evaluating the Reliability of LLMs against Jailbreak Attackscited by 8
  12. 2023Hazards from Increasingly Accessible Fine-Tuning of Downloadable Foundation Modelscited by 7
13 more
  1. 2023Towards Responsible Governance of Biological Design Toolscited by 5
  2. 2025Mapping AI Benchmark Data to Quantitative Risk Estimates Through Expert Elicitationcited by 4
  3. 2025Normative Conflicts and Shallow AI Alignmentcited by 4
  4. 2025Distinguishing Task-Specific and General-Purpose AI in Regulationcited by 3
  5. 2026Large Language Models Generate Harmful Responses Using a Distinct Mechanism, Shared Across Harm Typescited by 3
  6. 2025Supervision policies can shape long-term risk management in general-purpose AI modelscited by 2
  7. 2025Head-Specific Intervention Can Induce Misaligned AI Coordination in Large Language Modelscited by 1
  8. 2025Super Suffixes: Bypassing Text Generation Alignment and Guard Models Simultaneouslycited by 0
  9. 2026Jailbreaking LLMs Without Gradients or Priors: Effective and Transferable Attackscited by 0
  10. 2026Beyond Suffixes: Token Position in GCG Adversarial Attacks on Large Language Modelscited by 0
  11. 2026Refusal geometry reflects refusal training: diverse refusal prefixes can raise stable rank and weaken refusal vector ablation attackscited by 0
  12. 2026Artificial intelligence and biosecurity: capabilities, threat pathways, and defense-in-depth governancecited by 0
  13. 2026Auditing System-1 Models on Biosecurity-Relevant Benchmarks: Calibration, Selective Prediction, and Permutation Instability in a Non-Generative Modelcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025PropensityBench: Evaluating Latent Safety Risks in Large Language Models via an Agentic Approach
  2. 2026Epidemiology of Model Collapse: Modeling Synthetic Data Contamination via Bilayer SIR Dynamics
  3. 2026The Epi-LLM Framework: probing LLM behavioral priors through epidemiological agent-based models
  4. 2026Exposing the Systematic Vulnerability of Open-Weight Models to Prefill Attacks
  5. 2026Security in the Fine-Tuning Lifecycle of Large Language Models: Threats, Defenses,Evaluation, and Future Directions
  6. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safety
  7. 2026Coordinated Pandemic Control with Large Language Model Agents as Policymaking Assistants
  8. 2025Do Large Language Model Agents Exhibit a Survival Instinct? An Empirical Study in a Sugarscape-Style Simulation