# Adversarial Attack on Large Language Models using Exponentiated Gradient Descent

arXiv: https://arxiv.org/abs/2505.09820 · HTML: https://mlsanity.com/paper/2505.09820/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Jailbreaking, one of 12 directions in LLM safety and security.
- Builds on 39 of its 51 references that are in the map.
- First posted on arXiv 14 May 2025.

## Placed in

- LLM safety and security / [Jailbreaking](https://mlsanity.com/safety/jailbreaking/)

## Builds on (most cited first, 30 of 39)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [Computational Optimal Transport](https://mlsanity.com/paper/1803.00567/) (2018)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [Safe RLHF: Safe Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2310.12773/) (2023)
- [JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models](https://mlsanity.com/paper/2404.01318/) (2024)
- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation](https://mlsanity.com/paper/2310.06987/) (2023)
- [Unified Pre-training for Program Understanding and Generation](https://mlsanity.com/paper/2103.06333/) (2021)
- [Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery](https://mlsanity.com/paper/2302.03668/) (2023)
- [Are aligned neural networks adversarially aligned?](https://mlsanity.com/paper/2306.15447/) (2023)
- [Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models](https://mlsanity.com/paper/2307.14539/) (2023)
- [AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts](https://mlsanity.com/paper/2010.15980/) (2020)
- [Pretraining Language Models with Human Preferences](https://mlsanity.com/paper/2302.08582/) (2023)
- [AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs](https://mlsanity.com/paper/2404.07921/) (2024)
- [RAIN: Your Language Models Can Align Themselves without Finetuning](https://mlsanity.com/paper/2309.07124/) (2023)
- [Who is GPT-3? An Exploration of Personality, Values and Demographics](https://mlsanity.com/paper/2209.14338/) (2022)

## Built on by (most cited first, 4 of 4)

- [Universal and Transferable Adversarial Attack on Large Language Models Using Exponentiated Gradient Descent](https://mlsanity.com/paper/2508.14853/) (2025)
- [Efficient Extractive Text Summarization for Online News Articles Using Machine Learning](https://mlsanity.com/paper/2509.15614/) (2025)
- [On the Hardness of Junking LLMs](https://mlsanity.com/paper/2605.05116/) (2026)
- [ATTCK-R: Knowledge-Aware Long-Horizon Jailbreaking for Protected LLMs](https://mlsanity.com/paper/2610.16305/) (2026)

## Abstract (arXiv)

As Large Language Models (LLMs) are widely used, understanding them systematically is key to improving their safety and realizing their full potential. Although many models are aligned using techniques such as reinforcement learning from human feedback (RLHF), they are still vulnerable to jailbreaking attacks. Some of the existing adversarial attack methods search for discrete tokens that may jailbreak a target model while others try to optimize the continuous space represented by the tokens of the model's vocabulary. While techniques based on the discrete space may prove to be inefficient, optimization of continuous token embeddings requires projections to produce discrete tokens, which might render them ineffective. To fully utilize the constraints and the structures of the space, we develop an intrinsic optimization technique using exponentiated gradient descent with the Bregman projection method to ensure that the optimized one-hot encoding always stays within the probability simplex. We prove the convergence of the technique and implement an efficient algorithm that is effective in jailbreaking several widely used LLMs. We demonstrate the efficacy of the proposed technique using five open-source LLMs on four openly available datasets. The results show that the technique achieves a higher success rate with great efficiency compared to three other state-of-the-art jailbreaking techniques. The source code for our implementation is available at: https://github.com/sbamit/Exponentiated-Gradient-Descent-LLM-Attack