DROJ: A Prompt-Driven Attack against Large Language Models
On the map: LLM safety and security → Jailbreaking
Abstract (arXiv)
Large Language Models (LLMs) have demonstrated exceptional capabilities across various natural language processing tasks. Due to their training on internet-sourced datasets, LLMs can sometimes generate objectionable content, necessitating extensive alignment with human feedback to avoid such outputs. Despite massive alignment efforts, LLMs remain susceptible to adversarial jailbreak attacks, which usually are manipulated prompts designed to circumvent safety mechanisms and elicit harmful responses. Here, we introduce a novel approach, Directed Rrepresentation Optimization Jailbreak (DROJ), which optimizes jailbreak prompts at the embedding level to shift the hidden representations of harmful queries towards directions that are more likely to elicit affirmative responses from the model. Our evaluations on LLaMA-2-7b-chat model show that DROJ achieves a 100% keyword-based Attack Success Rate (ASR), effectively preventing direct refusals. However, the model occasionally produces repetitive and non-informative responses. To mitigate this, we introduce a helpfulness system prompt that enhances the utility of the model's responses. Our code is available at https://github.com/Leon-Leyang/LLM-Safeguard.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 19 of its 24 references that are in the map. First posted on arXiv 14 Nov 2024.
Builds on
19 of 24 references.
Closest ancestors · 9
Same direction, newest first
- 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation6 Nov
- 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation10 Oct
- 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
- 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
- 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Models4 Sep
- 2023Detecting Language Model Attacks with Perplexity27 Aug
- 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
- 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul
Other references · 10
- 2023GPT-4 Technical Reportcited by 18,396
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2017Deep reinforcement learning from human preferencescited by 3,912
- 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
- 2023Mistral 7Bcited by 3,164
- 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
- 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 244
- 2021Gradient-based Adversarial Attacks against Text Transformerscited by 234
- 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 199
Built on by
Citing papers per year: 2025: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2024Making Them Ask and Answer: Jailbreaking Large Language Models in Few Queries via Disguise and Reconstruction
- 2024Jailbreak Attacks and Defenses Against Large Language Models: A Survey
- 2024Jailbreaking and Mitigation of Vulnerabilities in Large Language Models
- 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editing
- 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily
- 2026Adversarial Reframing: A Framework for Targeted Generation in Language Models
- 2025LatentBreak: Jailbreaking Large Language Models through Latent Space Feedback
- 2025AutoAdv: Automated Adversarial Prompting for Multi-Turn Jailbreaking of Large Language Models