mlsanityData to 7 Oct 2026

Is the System Message Really Important to Jailbreaks in Large Language Models?

arXiv 2402.14857 ↗First posted 20 Feb 2024cs.CL cs.AI cs.CRcited by 19

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

The rapid evolution of Large Language Models (LLMs) has rendered them indispensable in modern society. While security measures are typically to align LLMs with human values prior to release, recent studies have unveiled a concerning phenomenon named "Jailbreak". This term refers to the unexpected and potentially harmful responses generated by LLMs when prompted with malicious questions. Most existing research focus on generating jailbreak prompts but system message configurations vary significantly in experiments. In this paper, we aim to answer a question: Is the system message really important for jailbreaks in LLMs? We conduct experiments in mainstream LLMs to generate jailbreak prompts with varying system messages: short, long, and none. We discover that different system messages have distinct resistances to jailbreaks. Therefore, we explore the transferability of jailbreaks across LLMs with different system messages. Furthermore, we propose the System Messages Evolutionary Algorithm (SMEA) to generate system messages that are more resistant to jailbreak prompts, even with minor changes. Through SMEA, we get a robust system messages population with little change in the length of system messages. Our research not only bolsters LLMs security but also raises the bar for jailbreaks, fostering advancements in this field of study.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 27 of its 53 references that are in the map. First posted on arXiv 20 Feb 2024.

Builds on

27 of 53 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  2. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation10 Oct
  3. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts19 Sep
  4. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Models4 Sep
  5. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models7 Aug
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  7. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul
  8. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study23 May

Other references · 19

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  8. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  9. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  10. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  11. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  12. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
7 more
  1. 2022Red Teaming Language Models with Language Modelscited by 983
  2. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  3. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
  4. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  5. 2023Are aligned neural networks adversarially aligned?cited by 305
  6. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  7. 2023Fairness-guided Few-shot Prompting for Large Language Modelscited by 53

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 11; 2026: 1.

19 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  2. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editingcited by 77
  3. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  4. 2024Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMscited by 47
  5. 2025A Closer Look at System Prompt Robustnesscited by 31
  6. 2024JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Modelscited by 28
  7. 2024SysBench: Can Large Language Models Follow System Messages?cited by 26
  8. 2024Adversarial Tuning: Defending Against Jailbreak Attacks for LLMscited by 23
  9. 2024Mission Impossible: A Statistical Perspective on Jailbreaking LLMscited by 21
  10. 2025Monitoring Decomposition Attacks in LLMs with Lightweight Sequential Monitorscited by 21
  11. 2024The VLLM Safety Paradox: Dual Ease in Jailbreak Attack and Defensecited by 20
  12. 2025From Judgment to Interference: Early Stopping LLM Harmful Outputs via Streaming Content Monitoringcited by 20
7 more
  1. 2025System Prompt Optimization with Meta-Learningcited by 18
  2. 2025E2AT: Multimodal Jailbreak Defense via Dynamic Joint Optimization for Multimodal Large Language Modelscited by 12
  3. 2025SoK: Trust-Authorization Mismatch in LLM Agent Interactionscited by 11
  4. 2026Knowledge-Driven Multi-Turn Jailbreaking on Large Language Modelscited by 6
  5. 2025SafeLLM: Unlearning Harmful Outputs from Large Language Models against Jailbreak Attackscited by 4
  6. 2025ReasoningGuard: Safeguarding Large Reasoning Models with Inference-time Safety Aha Momentscited by 2
  7. 2025SafeBehavior: Simulating Human-Like Multistage Reasoning to Mitigate Jailbreak Attacks in Large Language Modelscited by 0