# Multi-concept Model Immunization through Differentiable Model Merging

arXiv: https://arxiv.org/abs/2412.15320 · HTML: https://mlsanity.com/paper/2412.15320/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 53 references that are in the map.
- First posted on arXiv 19 Dec 2024.

## Builds on (most cited first, 2 of 2)

- [Mist: Towards Improved Adversarial Examples for Diffusion Models](https://mlsanity.com/paper/2305.12683/) (2023)
- [DFWLayer: Differentiable Frank-Wolfe Optimization Layer](https://mlsanity.com/paper/2308.10806/) (2023)

## Built on by (most cited first, 4 of 4)

- [Model Immunization from a Condition Number Perspective](https://mlsanity.com/paper/2505.23760/) (2025)
- [Knowledge Distillation Detection for Open-weights Models](https://mlsanity.com/paper/2510.02302/) (2025)
- [Coffee: Controllable Diffusion Fine-tuning](https://mlsanity.com/paper/2511.14113/) (2025)
- [Designing to Forget: Deep Semi-parametric Models for Unlearning](https://mlsanity.com/paper/2603.22870/) (2026)

## Abstract (arXiv)

Model immunization is an emerging direction that aims to mitigate the potential risk of misuse associated with open-sourced models and advancing adaptation methods. The idea is to make the released models' weights difficult to fine-tune on certain harmful applications, hence the name ``immunized''. Recent work on model immunization focuses on the single-concept setting. However, models need to be immunized against multiple concepts in real-world situations. To address this gap, we propose an immunization algorithm that, simultaneously, learns a single ``difficult initialization'' for adaptation methods over a set of concepts. We achieve this by incorporating a differentiable merging layer that combines a set of model weights adapted over multiple concepts. In our experiments, we demonstrate the effectiveness of multi-concept immunization by generalizing prior work's experiment setup of re-learning and personalization adaptation to multiple concepts.