mlsanityData to 7 Oct 2026

Merging Improves Self-Critique Against Jailbreak Attacks

arXiv 2406.07188 ↗First posted 11 Jun 2024cs.CL cs.AIcited by 8

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

The robustness of large language models (LLMs) against adversarial manipulations, such as jailbreak attacks, remains a significant challenge. In this work, we propose an approach that enhances the self-critique capability of the LLM and further fine-tunes it over sanitized synthetic data. This is done with the addition of an external critic model that can be merged with the original, thus bolstering self-critique capabilities and improving the robustness of the LLMs response to adversarial prompts. Our results demonstrate that the combination of merging and self-critique can reduce the attack success rate of adversaries significantly, thus offering a promising defense mechanism against jailbreak attacks. Code, data and models released at https://github.com/vicgalle/merging-self-critique-jailbreaks .

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 32 of its 37 references that are in the map. First posted on arXiv 11 Jun 2024.

Builds on

32 of 37 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Defending LLMs against Jailbreaking Attacks via Backtranslation26 Feb
  2. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  3. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models7 Aug
  4. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul
  5. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study23 May

Other references · 27

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  5. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  6. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2024Mixtral of Expertscited by 1,785
  9. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  10. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  11. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  12. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
15 more
  1. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  2. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  3. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  4. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  5. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 356
  6. 2023Are aligned neural networks adversarially aligned?cited by 305
  7. 2023Role-Play with Large Language Modelscited by 290
  8. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  9. 2023Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselvescited by 123
  10. 2023Exploiting Asymmetry for Synthetic Training Data Generation: SynthIE and the Case of Information Extractioncited by 89
  11. 2023TinyGSM: achieving >80% on GSM8k with small language modelscited by 68
  12. 2023It's MBR All the Way Down: Modern Generation Techniques Through the Lens of Minimum Bayes Riskcited by 49
  13. 2023Hijacking Large Language Models via Adversarial In-Context Learningcited by 41
  14. 2024Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignmentcited by 28
  15. 2023Distilled Self-Critique of LLMs with Synthetic Data: a Bayesian Perspectivecited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 4; 2026: 2.

8 papers.

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  2. 2025ThinkGuard: Deliberative Slow Thinking Leads to Cautious Guardrailscited by 29
  3. 2025Think Twice, Generate Once: Safeguarding by Progressive Self-Reflectioncited by 11
  4. 2024Recent Advances in Attack and Defense Approaches of Large Language Modelscited by 5
  5. 2025MetaSC: Test-Time Safety Specification Optimization for Language Modelscited by 4
  6. 2025A Systematic Study of In-the-Wild Model Merging for Large Language Modelscited by 4
  7. 2026Colluding LoRA: A Compositional Vulnerability in LLM Safety Alignmentcited by 2
  8. 2026When Skills Meet Safety: Benchmarking and Characterizing the Adaptive Jailbreak Robustness of Skill-Merged LLMscited by 0