# GRP-Obliteration: Unaligning LLMs With a Single Unlabeled Prompt

arXiv: https://arxiv.org/abs/2602.06258 · HTML: https://mlsanity.com/paper/2602.06258/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 32 of its 41 references that are in the map.
- First posted on arXiv 5 Feb 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Safety alignment and jailbreak attacks](https://mlsanity.com/alignment/safety-alignment-and-jailbreak-attacks/)
- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 30 of 32)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024)
- [Erasing Concepts from Diffusion Models](https://mlsanity.com/paper/2303.07345/) (2023)
- [Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models](https://mlsanity.com/paper/2211.05105/) (2022)
- [Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation](https://mlsanity.com/paper/2310.06987/) (2023)
- [Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack](https://mlsanity.com/paper/2404.01833/) (2024)
- [A StrongREJECT for Empty Jailbreaks](https://mlsanity.com/paper/2402.10260/) (2024)
- [Multi-step Jailbreaking Privacy Attacks on ChatGPT](https://mlsanity.com/paper/2304.05197/) (2023)
- [Large Language Model Alignment: A Survey](https://mlsanity.com/paper/2309.15025/) (2023)
- [Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs](https://mlsanity.com/paper/2502.17424/) (2025)
- [Reliable and Efficient Concept Erasure of Text-to-Image Diffusion Models](https://mlsanity.com/paper/2407.12383/) (2024)
- [Universal Jailbreak Backdoors from Poisoned Human Feedback](https://mlsanity.com/paper/2311.14455/) (2023)
- [What is in Your Safe Data? Identifying Benign Data that Breaks Safety](https://mlsanity.com/paper/2404.01099/) (2024)
- [Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey](https://mlsanity.com/paper/2409.18169/) (2024)
- [Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples](https://mlsanity.com/paper/2510.07192/) (2025)
- [SafeGen: Mitigating Sexually Explicit Content Generation in Text-to-Image Models](https://mlsanity.com/paper/2404.06666/) (2024)
- [Persistent Pre-Training Poisoning of LLMs](https://mlsanity.com/paper/2410.13722/) (2024)
- [T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generation](https://mlsanity.com/paper/2501.12612/) (2025)
- [Leveraging the Context through Multi-Round Interactions for Jailbreaking Attacks](https://mlsanity.com/paper/2402.09177/) (2024)
- [Is poisoning a real threat to LLM alignment? Maybe more so than you think](https://mlsanity.com/paper/2406.12091/) (2024)
- [AlignGuard: Scalable Safety Alignment for Text-to-Image Generation](https://mlsanity.com/paper/2412.10493/) (2024)
- [TwinBreak: Jailbreaking LLM Security Alignments based on Twin Prompts](https://mlsanity.com/paper/2506.07596/) (2025)
- [Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage Policy Optimization](https://mlsanity.com/paper/2412.18279/) (2024)

## Built on by (most cited first, 7 of 7)

- [Learning to Attack and Defend: Adaptive Red Teaming of Language Models via GRPO](https://mlsanity.com/paper/2606.09701/) (2026)
- [MOSAIC: Composable Safety Alignment with Modular Control Tokens](https://mlsanity.com/paper/2603.16210/) (2026)
- [Matryoshka attribution: Learning to attribute language model outputs to representations and weights](https://mlsanity.com/paper/2609.25518/) (2026)
- [Behind Harmful Compliance: Behavioral and Mechanistic Divergence Across LLM Jailbreaks](https://mlsanity.com/paper/2604.18510/) (2026)
- [Fool's Gold: Defensive Deception Against Safety-Removal Attacks on Open-Weight Models](https://mlsanity.com/paper/2608.17202/) (2026)
- [Divide, Consult, Conquer: Capability Laundering Through Aligned LLMs](https://mlsanity.com/paper/2609.15383/) (2026)
- [PatchBench: Measuring Collateral Damage in Activation Patching](https://mlsanity.com/paper/2610.10276/) (2026)

## Abstract (arXiv)

Safety alignment is only as robust as its weakest failure mode. Despite extensive work on safety post-training, it has been shown that models can be readily unaligned through post-deployment fine-tuning. However, these methods often require extensive data curation and degrade model utility. In this work, we extend the practical limits of unalignment by introducing GRP-Obliteration (GRP-Oblit), a method that uses Group Relative Policy Optimization (GRPO) to directly remove safety constraints from target models. We show that a single unlabeled prompt is sufficient to reliably unalign safety-aligned models while largely preserving their utility, and that GRP-Oblit achieves stronger unalignment on average than existing state-of-the-art techniques. Moreover, GRP-Oblit generalizes beyond language models and can also unalign diffusion-based image generation systems. We evaluate GRP-Oblit on six utility benchmarks and five safety benchmarks across fifteen 7-20B parameter models, spanning instruct and reasoning models, as well as dense and MoE architectures. The evaluated model families include GPT-OSS, distilled DeepSeek, Gemma, Llama, Ministral, and Qwen.