mlsanityData to 7 Oct 2026

Sockpuppetting: Jailbreaking LLMs by Combining Prefilling with Optimization

arXiv 2601.13359 ↗First posted 19 Jan 2026cs.CL cs.CR cs.LGcited by 0

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Prefill attacks are an effective and low-cost jailbreaking method, as they directly insert an acceptance sequence (e.g., "Sure, here is...") at the start of an LLM's output and lead the model to continue the response. We make two contributions to this prior work. First, we show that an unsophisticated adversary can improve the well-known prefill attacks by ensembling a small number of prefill variants. Running three easy-to-generate prefills yields a combined attack success rate (ASR) of 22%, 90%, and 99% on Gemma-7B, Llama-3.1-8B, and Qwen3-8B respectively, an up to 38 percentage point improvement over the standard "Sure, here's..." prefill and up to 82 percentage points over our reproduction of GCG (Zou et al., 2023). Second, we introduce "sockpuppetting", a hybrid attack that optimizes an adversarial suffix placed inside the "assistant" message block of the chat template, rather than within the user prompt. The rolling variant of this attack, RollingSockpuppetGCG, increases prompt-agnostic ASR by up to 64 percentage points over our universal GCG baseline on Llama-3.1-8B. An ablation indicates that part of this gain stems from the choice of acceptance sequence rather than suffix placement alone (Appendix F). Both findings highlight the need for defences against output-prefix injection in open-weight models. Code: https://gitlab.com/asendotsinski/sockpuppetting

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 42 of its 49 references that are in the map. First posted on arXiv 19 Jan 2026.

Builds on

42 of 49 references.

Closest ancestors · 23

Same direction, newest first

  1. 2025Prefill-level Jailbreak: A Black-Box Risk Analysis of Large Language Models28 Apr
  2. 2025Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming31 Jan
  3. 2024AdvPrefix: An Objective for Nuanced LLM Jailbreaks13 Dec
  4. 2024AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMs3 Oct
  5. 2024AdaPPA: Adaptive Position Pre-Fill Jailbreak Attack Approach Targeting LLMs11 Sep
  6. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs22 Jul
  7. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Survey5 Jul
  8. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs21 Apr
  9. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks2 Apr
  10. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models28 Mar
  11. 2024Fast Adversarial Attacks on Language Models In One GPU Minute23 Feb
  12. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability13 Feb
11 more
  1. 2023Bypassing the Safety Training of Open-Source LLMs with Priming Attacks19 Dec
  2. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models23 Oct
  3. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  4. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation10 Oct
  5. 2023Multilingual Jailbreak Challenges in Large Language Models10 Oct
  6. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  7. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts19 Sep
  8. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  9. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models7 Aug
  10. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  11. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul

Other references · 19

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2025Gemma 3 Technical Reportcited by 1,686
  8. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  9. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  10. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  11. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  12. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
7 more
  1. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  2. 2022A Survey of Machine Unlearningcited by 298
  3. 2023Adversarial Attacks and Defenses in Large Language Models: Old and New Threatscited by 49
  4. 2025Improving LLM Safety Alignment with Dual-Objective Optimizationcited by 40
  5. 2025Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMscited by 37
  6. 2023Can large language models democratize access to dual-use biotechnology?cited by 33
  7. 2025OpenAI's Approach to External Red Teaming for AI Models and Systemscited by 26

Built on by

Not cited yet.