# LLMs are Vulnerable to Malicious Prompts Disguised as Scientific Language

arXiv: https://arxiv.org/abs/2501.14073 · HTML: https://mlsanity.com/paper/2501.14073/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security.
- First posted on arXiv 23 Jan 2025.

## Placed in

- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Built on by (most cited first, 17 of 17)

- [Towards Scientific Intelligence: A Survey of LLM-based Scientific Agents](https://mlsanity.com/paper/2503.24047/) (2025)
- [From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discovery](https://mlsanity.com/paper/2508.14111/) (2025)
- [DREAMS: Density Functional Theory Based Research Engine for Agentic Materials Simulation](https://mlsanity.com/paper/2507.14267/) (2025)
- [A Technical Survey of Reinforcement Learning Techniques for Large Language Models](https://mlsanity.com/paper/2507.04136/) (2025)
- [Enhancing Model Defense Against Jailbreaks with Proactive Safety Reasoning](https://mlsanity.com/paper/2501.19180/) (2025)
- [Silenced Biases: The Dark Side LLMs Learned to Refuse](https://mlsanity.com/paper/2511.03369/) (2025)
- [Bullying the Machine: How Personas Increase LLM Vulnerability](https://mlsanity.com/paper/2505.12692/) (2025)
- [From Superficial Outputs to Superficial Learning: Risks of Large Language Models in Education](https://mlsanity.com/paper/2509.21972/) (2025)
- [XtraGPT: Context-Aware and Controllable Academic Paper Revision via Human-AI Collaboration](https://mlsanity.com/paper/2505.11336/) (2025)
- [CoBia: Constructed Conversations Can Trigger Otherwise Concealed Societal Biases in LLMs](https://mlsanity.com/paper/2510.09871/) (2025)
- [Taxonomy-Adaptive Moderation Model with Robust Guardrails for Large Language Models](https://mlsanity.com/paper/2512.05339/) (2025)
- [Multi-Agent Framework for Threat Mitigation and Resilience in AI-Based Systems](https://mlsanity.com/paper/2512.23132/) (2025)
- [Conflicts Make Large Reasoning Models Vulnerable to Attacks](https://mlsanity.com/paper/2604.09750/) (2026)
- [Fair outputs, Biased Internals: Causal Potency and Asymmetry of Latent Bias in LLMs for High-Stakes Decisions](https://mlsanity.com/paper/2605.15217/) (2026)
- [SEVRA-BENCH: Social Engineering of Vulnerabilities in Review Agents](https://mlsanity.com/paper/2606.13757/) (2026)
- [Malicious Repurposing of Open Science Artefacts by Using Large Language Models](https://mlsanity.com/paper/2601.18998/) (2026)
- [Metaphor Is Not All Attention Needs](https://mlsanity.com/paper/2605.12128/) (2026)

## Abstract (arXiv)

As large language models (LLMs) have been deployed in various real-world settings, concerns about the harm they may propagate have grown. Various jailbreaking techniques have been developed to expose the vulnerabilities of these models and improve their safety. This work reveals that many state-of-the-art LLMs are vulnerable to malicious requests hidden behind scientific language. Specifically, our experiments with GPT4o, GPT4o-mini, GPT-4, LLama3-405B-Instruct, Llama3-70B-Instruct, Cohere, Gemini models demonstrate that, the models' biases and toxicity substantially increase when prompted with requests that deliberately misinterpret social science and psychological studies as evidence supporting the benefits of stereotypical biases. Alarmingly, these models can also be manipulated to generate fabricated scientific arguments claiming that biases are beneficial, which can be used by ill-intended actors to systematically jailbreak these strong LLMs. Our analysis studies various factors that contribute to the models' vulnerabilities to malicious requests in academic language. Mentioning author names and venues enhances the persuasiveness of models, and the bias scores increase as dialogues progress. Our findings call for a more careful investigation on the use of scientific data for training LLMs.