mlsanityData to 14 Oct 2026

Jailbreaking LLMs via Calibration

arXiv 2602.00619 ↗First posted 31 Jan 2026cs.CL cs.AI cs.CRcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Safety alignment in Large Language Models (LLMs) often creates a systematic discrepancy between a model's aligned output and the underlying pre-aligned data distribution. We propose a framework in which the effect of safety alignment on next-token prediction is modeled as a systematic distortion of a pre-alignment distribution. We cast Weak-to-Strong Jailbreaking as a forecast aggregation problem and derive an optimal aggregation strategy characterized by a Gradient Shift in the loss-induced dual space. We show that logit-arithmetic jailbreaking methods are a special case of this framework under cross-entropy loss, and derive a broader family of aggregation rules corresponding to other proper losses. We also propose a new hybrid aggregation rule. Evaluations across red-teaming benchmarks and math utility tasks using frontier models demonstrate that our approach achieves superior Attack Success Rates and lower "Jailbreak Tax" compared with existing methods, especially on the safety-hardened gpt-oss-120b.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 43 of its 51 references that are in the map. First posted on arXiv 31 Jan 2026.

Builds on

43 of 51 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Speak Easy: Eliciting Harmful Jailbreaks from LLMs with Simple Interactions6 Feb
  2. 2024Mission Impossible: A Statistical Perspective on Jailbreaking LLMs2 Aug
  3. 2023Removing RLHF Protections in GPT-4 via Fine-Tuning9 Nov
  4. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  5. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 37

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2017On Calibration of Modern Neural Networkscited by 4,960
  5. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  6. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  7. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  8. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  9. 2022Red Teaming Language Models with Language Modelscited by 1,026
  10. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
  11. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
  12. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
25 more
  1. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
  2. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 441
  3. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 390
  4. 2024A StrongREJECT for Empty Jailbreakscited by 385
  5. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 367
  6. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 308
  7. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 278
  8. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 228
  9. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 214
  10. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 156
  11. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 107
  12. 2025H-CoT: Hijacking the Chain-of-Thought Safety Reasoning Mechanism to Jailbreak Large Reasoning Models, Including OpenAI o1/o3, DeepSeek-R1, and Gemini 2.0 Flash Thinkingcited by 99
  13. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 80
  14. 2024Chain of Attack: a Semantic-Driven Contextual Multi-Turn attacker for LLMcited by 77
  15. 2024Multi-Turn Context Jailbreak Attack on Large Language Models From First Principlescited by 56
  16. 2024Adversaries Can Misuse Combinations of Safe Modelscited by 32
  17. 2025Adversarial Reasoning at Jailbreaking Timecited by 29
  18. 2025The Jailbreak Tax: How Useful are Your Jailbreak Outputs?cited by 28
  19. 2024Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detectioncited by 23
  20. 2024Endless Jailbreaks with Bijection Learningcited by 19
  21. 2025Chain-of-Thought Hijackingcited by 16
  22. 2021Are You Smarter Than a Random Expert? The Robust Aggregation of Substitutable Signalscited by 13
  23. 2025LARGO: Latent Adversarial Reflection through Gradient Optimization for Jailbreaking LLMscited by 7
  24. 2021From Proper Scoring Rules to Max-Min Optimal Forecast Aggregationcited by 5
  25. 2025Strong but Brittle: Simple Attacks Subvert Reasoning-based Safety Guardrailscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep
  2. 2024LIAR: Leveraging Inference Time Alignment (Best-of-N) to Jailbreak LLMs in Seconds
  3. 2025A Domain-Based Taxonomy of Jailbreak Vulnerabilities in Large Language Models
  4. 2025Playing the Fool: Jailbreaking LLMs and Multimodal LLMs with Out-of-Distribution Strategy
  5. 2025Align to Misalign: Automatic LLM Jailbreak with Meta-Optimized LLM Judges
  6. 2024Improved Techniques for Optimization-Based Jailbreaking on Large Language Models
  7. 2024Jailbreaking Large Language Models Through Alignment Vulnerabilities in Out-of-Distribution Settings
  8. 2026Causal Front-Door Adjustment for Robust Jailbreak Attacks on LLMs