mlsanityData to 7 Oct 2026

AttnGCG: Enhancing Jailbreaking Attacks on LLMs with Attention Manipulation

arXiv 2410.09040 ↗First posted 11 Oct 2024cs.CLcited by 31

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

This paper studies the vulnerabilities of transformer-based Large Language Models (LLMs) to jailbreaking attacks, focusing specifically on the optimization-based Greedy Coordinate Gradient (GCG) strategy. We first observe a positive correlation between the effectiveness of attacks and the internal behaviors of the models. For instance, attacks tend to be less effective when models pay more attention to system prompts designed to ensure LLM safety alignment. Building on this discovery, we introduce an enhanced method that manipulates models' attention scores to facilitate LLM jailbreaking, which we term AttnGCG. Empirically, AttnGCG shows consistent improvements in attack efficacy across diverse LLMs, achieving an average increase of ~7% in the Llama-2 series and ~10% in the Gemma series. Our strategy also demonstrates robust attack transferability against both unseen harmful goals and black-box LLMs like GPT-3.5 and GPT-4. Moreover, we note our attention-score visualization is more interpretable, allowing us to gain better insights into how our targeted attention manipulation facilitates more effective jailbreaking. We release the code at https://github.com/UCSC-VLAA/AttnGCG-attack.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 31 of its 39 references that are in the map. First posted on arXiv 11 Oct 2024.

Builds on

31 of 39 references.

Closest ancestors · 17

Same direction, newest first

  1. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs11 Apr
  2. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks2 Apr
  3. 2024Attacking Large Language Models with Projected Gradient Descent14 Feb
  4. 2024All in How You Ask for It: Simple Black-Box Method for Jailbreak Attacks18 Jan
  5. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs12 Jan
  6. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automatically4 Dec
  7. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation6 Nov
  8. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models23 Oct
  9. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  10. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  11. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  12. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts19 Sep
5 more
  1. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Models4 Sep
  2. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  4. 2023MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots16 Jul
  5. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul

Other references · 14

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  9. 2024Mixtral of Expertscited by 1,785
  10. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  11. 2023How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMscited by 103
  12. 2023Attention-Enhancing Backdoor Attacks Against BERT-based Modelscited by 32
2 more
  1. 2023Sight Beyond Text: Multi-Modal Training Enhances LLMs in Truthfulness and Ethicscited by 14
  2. 2024AICAttack: Adversarial Image Captioning Attack with Attention-Based Optimizationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 16; 2026: 12.

31 papers.

Sort:

By area:

  1. 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasurescited by 93
  2. 2025STAR-1: Safer Alignment of Reasoning LLMs with 1K Datacited by 67
  3. 2025AgentSentinel: An End-to-End and Real-Time Security Defense Framework for Computer-Use Agentscited by 25
  4. 2024Exploiting the Index Gradients for Optimization-Based Jailbreaking on Large Language Modelscited by 15
  5. 2025D-Fusion: Direct Preference Optimization for Aligning Diffusion Models with Visually Consistent Samplescited by 14
  6. 2025May I have your Attention? Breaking Fine-Tuning based Prompt Injection Defenses using Architecture-Aware Attackscited by 14
  7. 2025Guiding not Forcing: Enhancing the Transferability of Jailbreaking Attacks on LLMs via Removing Superfluous Constraintscited by 11
  8. 2025Using Mechanistic Interpretability to Craft Adversarial Attacks against Large Language Modelscited by 9
  9. 2026HoneyTrap: Deceiving Large Language Model Attackers to Honeypot Traps with Resilient Multi-Agent Defensecited by 6
  10. 2024Model-Editing-Based Jailbreak against Safety-aligned Large Language Modelscited by 4
  11. 2025On Surjectivity of Neural Networks: Can you elicit any behavior from your model?cited by 4
  12. 2026Toward Universal and Transferable Jailbreak Attacks on Vision-Language Modelscited by 4
19 more
  1. 2026RouteHijack: Routing-Aware Attack on Mixture-of-Experts LLMscited by 4
  2. 2024Preventing Jailbreak Prompts as Malicious Tools for Cybercriminals: A Cyber Defense Perspectivecited by 3
  3. 2025When Good Sounds Go Adversarial: Jailbreaking Audio-Language Models with Benign Inputscited by 3
  4. 2026Safer by Diffusion, Broken by Context: Diffusion LLM's Safety Blessing and Its Failure Modecited by 3
  5. 2026Adversarial Attacks on Multimodal Large Language Models: A Comprehensive Surveycited by 3
  6. 2025Checkpoint-GCG: Auditing and Attacking Fine-Tuning-Based Prompt Injection Defensescited by 2
  7. 2025ReasoningGuard: Safeguarding Large Reasoning Models with Inference-time Safety Aha Momentscited by 2
  8. 2025NonTextual Target Attackcited by 2
  9. 2026Breaking MCP with Function Hijacking Attacks: Novel Threats for Function Calling and Agentic Modelscited by 2
  10. 2025Benign-to-Toxic Jailbreaking: Inducing Harmful Responses from Harmless Promptscited by 1
  11. 2025Eyes-on-Me: Scalable RAG Poisoning through Transferable Attention-Steering Attractorscited by 1
  12. 2026SoundBreak: A Systematic Study of Audio-Only Adversarial Attacks on Trimodal Modelscited by 1
  13. 2026SlotGCG: Exploiting the Positional Vulnerability in LLMs for Jailbreak Attackscited by 1
  14. 2025SafeBehavior: Simulating Human-Like Multistage Reasoning to Mitigate Jailbreak Attacks in Large Language Modelscited by 0
  15. 2025Distillability of LLM Security Logic: Predicting Attack Success Rate of Outline Filling Attack via Ranking Regressioncited by 0
  16. 2026Beyond Suffixes: Token Position in GCG Adversarial Attacks on Large Language Modelscited by 0
  17. 2026Sirens' Whisper: Inaudible Near-Ultrasonic Jailbreaks of Speech-Driven LLMscited by 0
  18. 2026Measuring the Wrong Thing: Internal Harmfulness Scores Anti-Rank Successful Jailbreakscited by 0
  19. 2026TACS: Trajectory-Aware Candidate Selection for LLM Jailbreak Suffix Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026TAO-Attack: Toward Advanced Optimization-Based Jailbreak Attacks for Large Language Models
  2. 2024Faster-GCG: Efficient Discrete Optimization Jailbreak Attacks against Aligned Large Language Models
  3. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editing
  4. 2024JailPO: A Novel Black-box Jailbreak Framework via Preference Optimization against Aligned LLMs
  5. 2026Circuit Discovery Helps Detect LLM Jailbreaking: A Mechanistic Interpretability Study
  6. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Survey
  7. 2025Attention Eclipse: Manipulating Attention to Bypass LLM Safety-Alignment
  8. 2025Formalization Driven LLM Prompt Jailbreaking via Reinforcement Learning