# Markpainting: Adversarial Machine Learning meets Inpainting

arXiv: https://arxiv.org/abs/2106.00660 · HTML: https://mlsanity.com/paper/2106.00660/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Watermarking generated content and models, one of 12 directions in LLM safety and security.
- Builds on 25 of its 42 references that are in the map.
- First posted on arXiv 1 Jun 2021.

## Placed in

- LLM safety and security / [Watermarking generated content and models](https://mlsanity.com/safety/watermarking-generated-content-and-models/)

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Practical Black-Box Attacks against Machine Learning](https://mlsanity.com/paper/1602.02697/) (2016)
- [Evasion Attacks against Machine Learning at Test Time](https://mlsanity.com/paper/1708.06131/) (2017)
- [ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models](https://mlsanity.com/paper/1708.03999/) (2017)
- [Generative Image Inpainting with Contextual Attention](https://mlsanity.com/paper/1801.07892/) (2018)
- [Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning](https://mlsanity.com/paper/1712.03141/) (2017)
- [NIPS 2016 Tutorial: Generative Adversarial Networks](https://mlsanity.com/paper/1701.00160/) (2016)
- [EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning](https://mlsanity.com/paper/1901.00212/) (2019)
- [Fast Face-swap Using Convolutional Neural Networks](https://mlsanity.com/paper/1611.09577/) (2016)
- [Sponge Examples: Energy-Latency Attacks on Neural Networks](https://mlsanity.com/paper/2006.03463/) (2020)
- [Recurrent Feature Reasoning for Image Inpainting](https://mlsanity.com/paper/2008.03737/) (2020)
- [Foreground-aware Image Inpainting](https://mlsanity.com/paper/1901.05945/) (2019)
- [Towards the Science of Security and Privacy in Machine Learning](https://mlsanity.com/paper/1611.03814/) (2016)
- [Image Inpainting via Generative Multi-column Convolutional Neural Networks](https://mlsanity.com/paper/1810.08771/) (2018)
- [Region Normalization for Image Inpainting](https://mlsanity.com/paper/1911.10375/) (2019)
- [Learning to Incorporate Structure Knowledge for Image Inpainting](https://mlsanity.com/paper/2002.04170/) (2020)
- [DeepGIN: Deep Generative Inpainting Network for Extreme Image Inpainting](https://mlsanity.com/paper/2008.07173/) (2020)
- [On Attribution of Deepfakes](https://mlsanity.com/paper/2008.09194/) (2020)
- [CR-Fill: Generative Image Inpainting with Auxiliary Contexutal Reconstruction](https://mlsanity.com/paper/2011.12836/) (2020)

## Built on by (most cited first, 9 of 9)

- [Watermark Vaccine: Adversarial Attacks to Prevent Watermark Removal](https://mlsanity.com/paper/2207.08178/) (2022)
- [Leveraging Adversarial Examples to Quantify Membership Information Leakage](https://mlsanity.com/paper/2203.09566/) (2022)
- [Learning to Immunize Images for Tamper Localization and Self-Recovery](https://mlsanity.com/paper/2210.15902/) (2022)
- [DRAW: Defending Camera-shooted RAW against Image Manipulation](https://mlsanity.com/paper/2307.16418/) (2023)
- [RWN: Robust Watermarking Network for Image Cropping Localization](https://mlsanity.com/paper/2110.05687/) (2021)
- [Diffusion LLMs are Natural Adversaries for any LLM](https://mlsanity.com/paper/2511.00203/) (2025)
- [Hiding Images into Images with Real-world Robustness](https://mlsanity.com/paper/2110.05689/) (2021)
- [Robust Watermarking for Video Forgery Detection with Improved Imperceptibility and Robustness](https://mlsanity.com/paper/2207.03409/) (2022)
- [Beyond Invisibility: Learning Robust Visible Watermarks for Stronger Copyright Protection](https://mlsanity.com/paper/2506.02665/) (2025)

## Abstract (arXiv)

Inpainting is a learned interpolation technique that is based on generative modeling and used to populate masked or missing pieces in an image; it has wide applications in picture editing and retouching. Recently, inpainting started being used for watermark removal, raising concerns. In this paper we study how to manipulate it using our markpainting technique. First, we show how an image owner with access to an inpainting model can augment their image in such a way that any attempt to edit it using that model will add arbitrary visible information. We find that we can target multiple different models simultaneously with our technique. This can be designed to reconstitute a watermark if the editor had been trying to remove it. Second, we show that our markpainting technique is transferable to models that have different architectures or were trained on different datasets, so watermarks created using it are difficult for adversaries to remove. Markpainting is novel and can be used as a manipulation alarm that becomes visible in the event of inpainting.