# REFIT: A Unified Watermark Removal Framework For Deep Learning Systems With Limited Data

arXiv: https://arxiv.org/abs/1911.07205 · HTML: https://mlsanity.com/paper/1911.07205/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Nov 2019.

## Built on by (most cited first, 30 of 44)

- [Adversarial Neuron Pruning Purifies Backdoored Deep Models](https://mlsanity.com/paper/2110.14430/) (2021)
- [Dataset Inference: Ownership Resolution in Machine Learning](https://mlsanity.com/paper/2104.10706/) (2021)
- [CATER: Intellectual Property Protection on Text Generation APIs via Conditional Watermarks](https://mlsanity.com/paper/2209.08773/) (2022)
- [Better Safe Than Sorry: Preventing Delusive Adversaries with Adversarial Training](https://mlsanity.com/paper/2102.04716/) (2021)
- [Non-Transferable Learning: A New Approach for Model Ownership Verification and Applicability Authorization](https://mlsanity.com/paper/2106.06916/) (2021)
- [Copyright Protection in Generative AI: A Technical Perspective](https://mlsanity.com/paper/2402.02333/) (2024)
- [Dataset Inference for Self-Supervised Models](https://mlsanity.com/paper/2209.09024/) (2022)
- [Towards Practical Deployment-Stage Backdoor Attack on Deep Neural Networks](https://mlsanity.com/paper/2111.12965/) (2021)
- [One-shot Neural Backdoor Erasing via Adversarial Weight Masking](https://mlsanity.com/paper/2207.04497/) (2022)
- [Data Hiding with Deep Learning: A Survey Unifying Digital Watermarking and Steganography](https://mlsanity.com/paper/2107.09287/) (2021)
- [What Doesn't Kill You Makes You Robust(er): How to Adversarially Train against Data Poisoning](https://mlsanity.com/paper/2102.13624/) (2021)
- [Rethinking White-Box Watermarks on Deep Learning Models under Neural Structural Obfuscation](https://mlsanity.com/paper/2303.09732/) (2023)
- [Blind Image Decomposition](https://mlsanity.com/paper/2108.11364/) (2021)
- [Adversarial Patch Attacks and Defences in Vision-Based Tasks: A Survey](https://mlsanity.com/paper/2206.08304/) (2022)
- [Model Barrier: A Compact Un-Transferable Isolation Domain for Model Intellectual Property Protection](https://mlsanity.com/paper/2303.11078/) (2023)
- [Yuan: Yielding Unblemished Aesthetics Through A Unified Network for Visual Imperfections Removal in Generated Images](https://mlsanity.com/paper/2501.08505/) (2025)
- [On the Effectiveness of Dataset Watermarking in Adversarial Settings](https://mlsanity.com/paper/2202.12506/) (2022)
- [Identifying Appropriate Intellectual Property Protection Mechanisms for Machine Learning Models: A Systematization of Watermarking, Fingerprinting, Model Access, and Attacks](https://mlsanity.com/paper/2304.11285/) (2023)
- [Backdoor Attacks on Time Series: A Generative Approach](https://mlsanity.com/paper/2211.07915/) (2022)
- [Say No to Freeloader: Protecting Intellectual Property of Your Deep Model](https://mlsanity.com/paper/2408.13161/) (2024)
- [Detect and remove watermark in deep neural networks via generative adversarial networks](https://mlsanity.com/paper/2106.08104/) (2021)
- [Evaluating the Robustness of Trigger Set-Based Watermarks Embedded in Deep Neural Networks](https://mlsanity.com/paper/2106.10147/) (2021)
- [Cracking White-box DNN Watermarks via Invariant Neuron Transforms](https://mlsanity.com/paper/2205.00199/) (2022)
- [Unsupervised Non-transferable Text Classification](https://mlsanity.com/paper/2210.12651/) (2022)
- [RemovalNet: DNN Fingerprint Removal Attacks](https://mlsanity.com/paper/2308.12319/) (2023)
- [MAP: MAsk-Pruning for Source-Free Model Intellectual Property Protection](https://mlsanity.com/paper/2403.04149/) (2024)
- [Ownership Verification of DNN Architectures via Hardware Cache Side Channels](https://mlsanity.com/paper/2102.03523/) (2021)
- [TATTOOED: A Robust Deep Neural Network Watermarking Scheme based on Spread-Spectrum Channel Coding](https://mlsanity.com/paper/2202.06091/) (2022)
- [Attention Distraction: Watermark Removal Through Continual Learning with Selective Forgetting](https://mlsanity.com/paper/2204.01934/) (2022)
- [Backdoor Attacks on Bayesian Neural Networks using Reverse Distribution](https://mlsanity.com/paper/2205.09167/) (2022)

## Abstract (arXiv)

Training deep neural networks from scratch could be computationally expensive and requires a lot of training data. Recent work has explored different watermarking techniques to protect the pre-trained deep neural networks from potential copyright infringements. However, these techniques could be vulnerable to watermark removal attacks. In this work, we propose REFIT, a unified watermark removal framework based on fine-tuning, which does not rely on the knowledge of the watermarks, and is effective against a wide range of watermarking schemes. In particular, we conduct a comprehensive study of a realistic attack scenario where the adversary has limited training data, which has not been emphasized in prior work on attacks against watermarking schemes. To effectively remove the watermarks without compromising the model functionality under this weak threat model, we propose two techniques that are incorporated into our fine-tuning framework: (1) an adaption of the elastic weight consolidation (EWC) algorithm, which is originally proposed for mitigating the catastrophic forgetting phenomenon; and (2) unlabeled data augmentation (AU), where we leverage auxiliary unlabeled data from other sources. Our extensive evaluation shows the effectiveness of REFIT against diverse watermark embedding schemes. In particular, both EWC and AU significantly decrease the amount of labeled training data needed for effective watermark removal, and the unlabeled data samples used for AU do not necessarily need to be drawn from the same distribution as the benign data for model evaluation. The experimental results demonstrate that our fine-tuning based watermark removal attacks could pose real threats to the copyright of pre-trained models, and thus highlight the importance of further investigating the watermarking problem and proposing more robust watermark embedding schemes against the attacks.