# RMCBench: Benchmarking Large Language Models' Resistance to Malicious Code

arXiv: https://arxiv.org/abs/2409.15154 · HTML: https://mlsanity.com/paper/2409.15154/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Jailbreaking, one of 12 directions in LLM safety and security.
- Builds on 39 of its 66 references that are in the map.
- First posted on arXiv 23 Sep 2024.

## Placed in

- LLM safety and security / [Jailbreaking](https://mlsanity.com/safety/jailbreaking/)

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://mlsanity.com/paper/2307.04657/) (2023)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts](https://mlsanity.com/paper/2309.10253/) (2023)
- [DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models](https://mlsanity.com/paper/2306.11698/) (2023)
- [Zephyr: Direct Distillation of LM Alignment](https://mlsanity.com/paper/2310.16944/) (2023)
- [Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study](https://mlsanity.com/paper/2305.13860/) (2023)
- [DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation](https://mlsanity.com/paper/2211.11501/) (2022)
- [Large Language Model Alignment: A Survey](https://mlsanity.com/paper/2309.15025/) (2023)
- [ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversation](https://mlsanity.com/paper/2310.17389/) (2023)
- [Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2](https://mlsanity.com/paper/2311.10702/) (2023)
- [Evaluating the Moral Beliefs Encoded in LLMs](https://mlsanity.com/paper/2307.14324/) (2023)
- [CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion](https://mlsanity.com/paper/2310.11248/) (2023)
- [ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation](https://mlsanity.com/paper/2308.01861/) (2023)
- [Multi-lingual Evaluation of Code Generation Models](https://mlsanity.com/paper/2210.14868/) (2022)
- [GitHub Copilot AI pair programmer: Asset or Liability?](https://mlsanity.com/paper/2206.15331/) (2022)
- [Few-shot training LLMs for project-specific code-summarization](https://mlsanity.com/paper/2207.04237/) (2022)
- [Towards an Understanding of Large Language Models in Software Engineering Tasks](https://mlsanity.com/paper/2308.11396/) (2023)

## Built on by (most cited first, 28 of 28)

- [Beyond the Protocol: Unveiling Attack Vectors in the Model Context Protocol (MCP) Ecosystem](https://mlsanity.com/paper/2506.02040/) (2025)
- [PurpCode: Reasoning for Safer Code Generation](https://mlsanity.com/paper/2507.19060/) (2025)
- [Large Language Model (LLM) for Software Security: Code Analysis, Malware Analysis, Reverse Engineering](https://mlsanity.com/paper/2504.07137/) (2025)
- [An Empirical Study of Agent Developer Practices in AI Agent Frameworks](https://mlsanity.com/paper/2512.01939/) (2025)
- [Top General Performance = Top Domain Performance? DomainCodeBench: A Multi-domain Code Generation Benchmark](https://mlsanity.com/paper/2412.18573/) (2024)
- [Exploring the Security Threats of Knowledge Base Poisoning in Retrieval-Augmented Code Generation](https://mlsanity.com/paper/2502.03233/) (2025)
- [LLMs Caught in the Crossfire: Malware Requests and Jailbreak Challenges](https://mlsanity.com/paper/2506.10022/) (2025)
- [RedCodeAgent: Automatic Red-teaming Agent against Diverse Code Agents](https://mlsanity.com/paper/2510.02609/) (2025)
- [MOCHA: Are Code Language Models Robust Against Multi-Turn Malicious Coding Prompts?](https://mlsanity.com/paper/2507.19598/) (2025)
- [Breaking the Code: Security Assessment of AI Code Agents Through Systematic Jailbreaking Attacks](https://mlsanity.com/paper/2510.01359/) (2025)
- [AlignCoder: Aligning Retrieval with Target Intent for Repository-Level Code Completion](https://mlsanity.com/paper/2601.19697/) (2026)
- [Smoke and Mirrors: Jailbreaking LLM-based Code Generation via Implicit Malicious Prompts](https://mlsanity.com/paper/2503.17953/) (2025)
- [SeCodePLT: A Unified Platform for Evaluating the Security of Code GenAI](https://mlsanity.com/paper/2410.11096/) (2024)
- [Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Quality](https://mlsanity.com/paper/2503.05860/) (2025)
- [DRAINCODE: Stealthy Energy Consumption Attacks on Retrieval-Augmented Code Generation via Context Poisoning](https://mlsanity.com/paper/2601.20615/) (2026)
- [PhoenixRepair: Rethinking Repair Strategy Exploration in Software Agents](https://mlsanity.com/paper/2607.18859/) (2026)
- [Web3 x AI Agents: Landscape, Integrations, and Foundational Challenges](https://mlsanity.com/paper/2508.02773/) (2025)
- [BlueCodeAgent: A Blue Teaming Agent Enabled by Automated Red Teaming for CodeGen AI](https://mlsanity.com/paper/2510.18131/) (2025)
- [Exploring the Security Threats of Retriever Backdoors in Retrieval-Augmented Code Generation](https://mlsanity.com/paper/2512.21681/) (2025)
- [Overlooked Safety Vulnerability in LLMs: Malicious Intelligent Optimization Algorithm Request and its Jailbreak](https://mlsanity.com/paper/2601.00213/) (2026)
- [Refusal Evaluation in Coding LLMs and Code Agents: A Systematic Review of Thirteen Malicious-Code Prompt Corpora (2023-2025)](https://mlsanity.com/paper/2605.20351/) (2026)
- [Amazon Nova AI Challenge -- Trusted AI: Advancing secure, AI-assisted software development](https://mlsanity.com/paper/2508.10108/) (2025)
- [Designing Empirical Studies on LLM-Based Code Generation: Towards a Reference Framework](https://mlsanity.com/paper/2510.03862/) (2025)
- [JPU: Bridging Jailbreak Defense and Unlearning via On-Policy Path Rectification](https://mlsanity.com/paper/2601.03005/) (2026)
- [Code as a Weapon: A Consensus-Labeled Prompt Bank for Measuring Coding-Model Compliance with Malicious-Code Requests](https://mlsanity.com/paper/2605.28734/) (2026)
- [Grammar-Constrained Decoding Can Jailbreak LLMs into Generating Malicious Code](https://mlsanity.com/paper/2606.11817/) (2026)
- [CS-Guard: Benchmarking LLM Guardrails for Code Generation Security](https://mlsanity.com/paper/2609.09798/) (2026)
- [Generalization Is Stability, Not Accuracy: Multi-Axis Evaluation of LLMs](https://mlsanity.com/paper/2610.01428/) (2026)

## Abstract (arXiv)

The emergence of Large Language Models (LLMs) has significantly influenced various aspects of software development activities. Despite their benefits, LLMs also pose notable risks, including the potential to generate harmful content and being abused by malicious developers to create malicious code. Several previous studies have focused on the ability of LLMs to resist the generation of harmful content that violates human ethical standards, such as biased or offensive content. However, there is no research evaluating the ability of LLMs to resist malicious code generation. To fill this gap, we propose RMCBench, the first benchmark comprising 473 prompts designed to assess the ability of LLMs to resist malicious code generation. This benchmark employs two scenarios: a text-to-code scenario, where LLMs are prompted with descriptions to generate code, and a code-to-code scenario, where LLMs translate or complete existing malicious code. Based on RMCBench, we conduct an empirical study on 11 representative LLMs to assess their ability to resist malicious code generation. Our findings indicate that current LLMs have a limited ability to resist malicious code generation with an average refusal rate of 40.36% in text-to-code scenario and 11.52% in code-to-code scenario. The average refusal rate of all LLMs in RMCBench is only 28.71%; ChatGPT-4 has a refusal rate of only 35.73%. We also analyze the factors that affect LLMs' ability to resist malicious code generation and provide implications for developers to enhance model robustness.