# Malicious Repurposing of Open Science Artefacts by Using Large Language Models

arXiv: https://arxiv.org/abs/2601.18998 · HTML: https://mlsanity.com/paper/2601.18998/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 40 references that are in the map.
- First posted on arXiv 26 Jan 2026.

## Builds on (most cited first, 17 of 17)

- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024)
- [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://mlsanity.com/paper/2307.04657/) (2023)
- [Safety Alignment Should Be Made More Than Just a Few Tokens Deep](https://mlsanity.com/paper/2406.05946/) (2024)
- [Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers](https://mlsanity.com/paper/2409.04109/) (2024)
- [Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonable](https://mlsanity.com/paper/2503.00555/) (2025)
- [Just How Toxic is Data Poisoning? A Unified Benchmark for Backdoor and Data Poisoning Attacks](https://mlsanity.com/paper/2006.12557/) (2020)
- [Chain of Ideas: Revolutionizing Research Via Novel Idea Development with LLM Agents](https://mlsanity.com/paper/2410.13185/) (2024)
- [On Protecting the Data Privacy of Large Language Models (LLMs): A Survey](https://mlsanity.com/paper/2403.05156/) (2024)
- [Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Models](https://mlsanity.com/paper/2403.17336/) (2024)
- [Learning diverse attacks on large language models for robust red-teaming and safety tuning](https://mlsanity.com/paper/2405.18540/) (2024)
- [Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2508.06709/) (2025)
- [LabSafety Bench: Benchmarking LLMs on Safety Issues in Scientific Labs](https://mlsanity.com/paper/2410.14182/) (2024)
- [A Vision for Auto Research with LLM Agents](https://mlsanity.com/paper/2504.18765/) (2025)
- [LLMs can Realize Combinatorial Creativity: Generating Creative Ideas via LLMs for Scientific Research](https://mlsanity.com/paper/2412.14141/) (2024)
- [LLMs are Vulnerable to Malicious Prompts Disguised as Scientific Language](https://mlsanity.com/paper/2501.14073/) (2025)
- [DiveR-CT: Diversity-enhanced Red Teaming Large Language Model Assistants with Relaxing Constraints](https://mlsanity.com/paper/2405.19026/) (2024)
- [SMILES-Prompting: A Novel Approach to LLM Jailbreak Attacks in Chemical Synthesis](https://mlsanity.com/paper/2410.15641/) (2024)

## Abstract (arXiv)

The rapid evolution of large language models (LLMs) has fuelled enthusiasm about their role in advancing scientific discovery, with studies exploring LLMs that autonomously generate and evaluate novel research ideas. However, little attention has been given to the possibility that such models could be exploited to produce harmful research by repurposing open science artefacts for malicious ends. We fill the gap by introducing an end-to-end pipeline that first bypasses LLM safeguards through persuasion-based jailbreaking, then reinterprets NLP papers to identify and repurpose their artefacts (datasets, methods, and tools) by exploiting their vulnerabilities, and finally assesses the safety of these proposals using our evaluation framework across three dimensions: harmfulness, feasibility of misuse, and soundness of technicality. Overall, our findings demonstrate that LLMs can generate harmful proposals by repurposing ethically designed open artefacts; however, we find that LLMs acting as evaluators strongly disagree with one another on evaluation outcomes: GPT-4.1 assigns higher scores (indicating greater potential harms, higher soundness and feasibility of misuse), Gemini-2.5-pro is markedly stricter, and Grok-3 falls between these extremes. This indicates that LLMs cannot yet serve as reliable judges in a malicious evaluation setup, making human evaluation essential for credible dual-use risk assessment.