# Can large language models democratize access to dual-use biotechnology?

arXiv: https://arxiv.org/abs/2306.03809 · HTML: https://mlsanity.com/paper/2306.03809/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 30 references that are in the map.
- First posted on arXiv 6 Jun 2023.

## Builds on (most cited first, 5 of 5)

- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks](https://mlsanity.com/paper/2302.05733/) (2023)
- [Emergent autonomous scientific research capabilities of large language models](https://mlsanity.com/paper/2304.05332/) (2023)

## Built on by (most cited first, 30 of 35)

- [An Overview of Catastrophic AI Risks](https://mlsanity.com/paper/2306.12001/) (2023)
- [AI Deception: A Survey of Examples, Risks, and Potential Solutions](https://mlsanity.com/paper/2308.14752/) (2023)
- [AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathways](https://mlsanity.com/paper/2406.02630/) (2024)
- [Black-Box Access is Insufficient for Rigorous AI Audits](https://mlsanity.com/paper/2401.14446/) (2024)
- [Frontier AI Regulation: Managing Emerging Risks to Public Safety](https://mlsanity.com/paper/2307.03718/) (2023)
- [Visibility into AI Agents](https://mlsanity.com/paper/2401.13138/) (2024)
- [Large Language Model Safety: A Holistic Survey](https://mlsanity.com/paper/2412.17686/) (2024)
- [Managing extreme AI risks amid rapid progress](https://mlsanity.com/paper/2310.17688/) (2023)
- [Hoodwinked: Deception and Cooperation in a Text-Based Game for Language Models](https://mlsanity.com/paper/2308.01404/) (2023)
- [A Safe Harbor for AI Evaluation and Red Teaming](https://mlsanity.com/paper/2403.04893/) (2024)
- [BadLlama: cheaply removing safety fine-tuning from Llama 2-Chat 13B](https://mlsanity.com/paper/2311.00117/) (2023)
- [Adversaries Can Misuse Combinations of Safe Models](https://mlsanity.com/paper/2406.14595/) (2024)
- [Open-Sourcing Highly Capable Foundation Models: An evaluation of risks, benefits, and alternative methods for pursuing open-source objectives](https://mlsanity.com/paper/2311.09227/) (2023)
- [Will releasing the weights of future large language models grant widespread access to pandemic agents?](https://mlsanity.com/paper/2310.18233/) (2023)
- [Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy and Research](https://mlsanity.com/paper/2412.06966/) (2024)
- [Hazards from Increasingly Accessible Fine-Tuning of Downloadable Foundation Models](https://mlsanity.com/paper/2312.14751/) (2023)
- [Measuring Mid-2025 LLM-Assistance on Novice Performance in Biology](https://mlsanity.com/paper/2602.16703/) (2026)
- [Towards Responsible Governance of Biological Design Tools](https://mlsanity.com/paper/2311.15936/) (2023)
- [To Err is AI : A Case Study Informing LLM Flaw Reporting Practices](https://mlsanity.com/paper/2410.12104/) (2024)
- [Mapping AI Benchmark Data to Quantitative Risk Estimates Through Expert Elicitation](https://mlsanity.com/paper/2503.04299/) (2025)
- [Misalignment or misuse? The AGI alignment tradeoff](https://mlsanity.com/paper/2506.03755/) (2025)
- [Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems](https://mlsanity.com/paper/2410.23472/) (2024)
- [Supervision policies can shape long-term risk management in general-purpose AI models](https://mlsanity.com/paper/2501.06137/) (2025)
- [A Survey on Training-free Alignment of Large Language Models](https://mlsanity.com/paper/2508.09016/) (2025)
- [A Blind Spot in Alignment: Quantifying Biosecurity Risks in Large Language Models](https://mlsanity.com/paper/2608.02684/) (2026)
- [CodeUnlearn: Amortized Zero-Shot Machine Unlearning in Language Models Using Discrete Concept](https://mlsanity.com/paper/2410.10866/) (2024)
- [Can Large Language Models Design Biological Weapons? Evaluating Moremi Bio](https://mlsanity.com/paper/2505.17154/) (2025)
- [David vs. Goliath: Verifiable Agent-to-Agent Jailbreaking via Reinforcement Learning](https://mlsanity.com/paper/2602.02395/) (2026)
- [RefusalBench: Why Refusal Rate Misranks Frontier LLMs on Biological Research Prompts](https://mlsanity.com/paper/2605.21545/) (2026)
- [Measuring Biological Capabilities and Risks of AI Agents](https://mlsanity.com/paper/2606.19899/) (2026)

## Abstract (arXiv)

Large language models (LLMs) such as those embedded in 'chatbots' are accelerating and democratizing research by providing comprehensible information and expertise from many different fields. However, these models may also confer easy access to dual-use technologies capable of inflicting great harm. To evaluate this risk, the 'Safeguarding the Future' course at MIT tasked non-scientist students with investigating whether LLM chatbots could be prompted to assist non-experts in causing a pandemic. In one hour, the chatbots suggested four potential pandemic pathogens, explained how they can be generated from synthetic DNA using reverse genetics, supplied the names of DNA synthesis companies unlikely to screen orders, identified detailed protocols and how to troubleshoot them, and recommended that anyone lacking the skills to perform reverse genetics engage a core facility or contract research organization. Collectively, these results suggest that LLMs will make pandemic-class agents widely accessible as soon as they are credibly identified, even to people with little or no laboratory training. Promising nonproliferation measures include pre-release evaluations of LLMs by third parties, curating training datasets to remove harmful concepts, and verifiably screening all DNA generated by synthesis providers or used by contract research organizations and robotic cloud laboratories to engineer organisms or viruses.