mlsanityData to 7 Oct 2026

Overlooked Safety Vulnerability in LLMs: Malicious Intelligent Optimization Algorithm Request and its Jailbreak

arXiv 2601.00213 ↗First posted 1 Jan 2026cs.CR cs.CLcited by 1

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

The widespread deployment of large language models (LLMs) has raised growing concerns about their misuse risks and associated safety issues. While prior studies have examined the safety of LLMs in general usage, code generation, and agent-based applications, their vulnerabilities in automated algorithm design remain underexplored. To fill this gap, this study investigates this overlooked safety vulnerability, with a particular focus on intelligent optimization algorithm design, given its prevalent use in complex decision-making scenarios. We introduce MalOptBench, a benchmark consisting of 60 malicious optimization algorithm requests, and propose MOBjailbreak, a jailbreak method tailored for this scenario. Through extensive evaluation of 13 mainstream LLMs including the latest GPT-5 and DeepSeek-V3.1, we reveal that most models remain highly susceptible to such attacks, with an average attack success rate of 83.59% and an average harmfulness score of 4.28 out of 5 on original harmful prompts, and near-complete failure under MOBjailbreak. Furthermore, we assess state-of-the-art plug-and-play defenses that can be applied to closed-source models, and find that they are only marginally effective against MOBjailbreak and prone to exaggerated safety behaviors. These findings highlight the urgent need for stronger alignment techniques to safeguard LLMs against misuse in algorithm design.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 25 of its 41 references that are in the map. First posted on arXiv 1 Jan 2026.

Builds on

25 of 41 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025LLMs Caught in the Crossfire: Malware Requests and Jailbreak Challenges9 Jun
  2. 2025Why Not Act on What You Know? Unleashing Safety Potential of LLMs via Self-Aware Guard Enhancement17 May
  3. 2025Rewrite to Jailbreak: Discover Learnable and Transferable Implicit Harmfulness Instruction16 Feb
  4. 2024RMCBench: Benchmarking Large Language Models' Resistance to Malicious Code23 Sep
  5. 2024DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakers25 Feb
  6. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding14 Feb
  7. 2024All in How You Ask for It: Simple Black-Box Method for Jailbreak Attacks18 Jan
  8. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automatically4 Dec
  9. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily14 Nov
  10. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreaker6 Nov
  11. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  12. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
3 more
  1. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts19 Sep
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  3. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts7 Jun

Other references · 10

  1. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  2. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  3. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 280
  4. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 124
  5. 2024LLaMEA: A Large Language Model Evolutionary Algorithm for Automatically Generating Metaheuristicscited by 92
  6. 2024A Systematic Survey on Large Language Models for Algorithm Designcited by 65
  7. 2025CoCoEvo: Co-Evolution of Programs and Test Cases to Enhance Code Generationcited by 14
  8. 2025BLADE: Benchmark suite for LLM-driven Automated Design and Evolution of iterative optimisation heuristicscited by 9
  9. 2025ParetoHqD: Fast Offline Multiobjective Alignment of Large Language Models using Pareto High-quality Datacited by 4
  10. 2021Understanding Underground Incentivized Review Servicescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026AutoSG: LLM-Driven Solver Generation Solely from Task Prompts for Expensive Optimizationcited by 0