mlsanityData to 14 Oct 2026

What Do Safety-Aligned LLMs Learn From Mixed Compliance Demonstrations?

arXiv 2606.20508 ↗First posted 18 Jun 2026cs.AI cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Prior work has shown that in-context demonstrations can jailbreak language models, but it remains unclear how models interpret different types of compliance demonstrations. We study this by mixing benign compliance demonstrations (non-harmful request, helpful response) with harmful compliance demonstrations (harmful request, helpful response) and testing three hypotheses about how demonstration composition drives harmful compliance. Across four models, we find that benign and harmful demonstrations are not interchangeable: benign demonstrations can either reduce or increase harmful compliance depending on the model. We further show that preference optimization is the critical training stage that prevents benign demonstrations from increasing harmful compliance, that demonstration ordering exhibits strong recency bias, and that models differ in how refusal interacts with in-context learning: some adopt demonstrated formatting even when refusing, while others override all in-context signals upon refusal. Taken together, this work moves beyond showing that demonstration-based jailbreaking works to characterizing how it works: what models extract from compliance demonstrations depends on demonstration content, ordering, and training methodology.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 17 of its 18 references that are in the map. First posted on arXiv 18 Jun 2026.

Builds on

17 of 18 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025PANDAS: Improving Many-shot Jailbreaking via Positive Affirmation, Negative Demonstration, and Adaptive Sampling4 Feb
  2. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal20 Jun
  3. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Models31 May

Other references · 14

  1. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  2. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  3. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  4. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  5. 2023Steering Language Models With Activation Engineeringcited by 922
  6. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
  7. 2023In-Context Learning Creates Task Vectorscited by 312
  8. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attackscited by 230
  9. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 161
  10. 2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
  11. 2023Overthinking the Truth: Understanding how Language Models Process False Demonstrationscited by 81
  12. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 70
2 more
  1. 2024PICLe: Eliciting Diverse Behaviors from Large Language Models with Persona In-Context Learningcited by 29
  2. 2025Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steeringcited by 29

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026The Refusal--Compliance Tradeoff: A Large-Scale Safety Behavior Audit of Large Language Models
  2. 2026Behind Harmful Compliance: Behavioral and Mechanistic Divergence Across LLM Jailbreaks
  3. 2024Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Training
  4. 2026LLMs Can Unlearn Refusal with Only 1,000 Benign Samples
  5. 2025Uncovering the Persuasive Fingerprint of LLMs in Jailbreaking Attacks
  6. 2026Deactivating Refusal Triggers: Understanding and Mitigating Overrefusal in Safety Alignment
  7. 2026Do Language Models Know When They'll Refuse? Probing Introspective Awareness of Safety Boundaries
  8. 2025Think Before Refusal : Triggering Safety Reflection in LLMs to Mitigate False Refusal Behavior