# MAGIC: Mask-Guided Image Synthesis by Inverting a Quasi-Robust Classifier

arXiv: https://arxiv.org/abs/2209.11549 · HTML: https://mlsanity.com/paper/2209.11549/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 46 references that are in the map.
- First posted on arXiv 23 Sep 2022.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Certified Adversarial Robustness via Randomized Smoothing](https://mlsanity.com/paper/1902.02918/) (2019)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Robustness May Be at Odds with Accuracy](https://mlsanity.com/paper/1805.12152/) (2018)
- [Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion](https://mlsanity.com/paper/1912.08795/) (2019)
- [Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning](https://mlsanity.com/paper/1712.03141/) (2017)
- [Understanding Deep Image Representations by Inverting Them](https://mlsanity.com/paper/1412.0035/) (2014)
- [SinGAN: Learning a Generative Model from a Single Natural Image](https://mlsanity.com/paper/1905.01164/) (2019)
- [Do Adversarially Robust ImageNet Models Transfer Better?](https://mlsanity.com/paper/2007.08489/) (2020)
- [Learning from Simulated and Unsupervised Images through Adversarial Training](https://mlsanity.com/paper/1612.07828/) (2016)
- [SEAN: Image Synthesis with Semantic Region-Adaptive Normalization](https://mlsanity.com/paper/1911.12861/) (2019)
- [Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks](https://mlsanity.com/paper/1604.04382/) (2016)
- [Visualizing Deep Convolutional Neural Networks Using Natural Pre-Images](https://mlsanity.com/paper/1512.02017/) (2015)
- [Interpreting Adversarially Trained Convolutional Neural Networks](https://mlsanity.com/paper/1905.09797/) (2019)
- [Local Class-Specific and Global Image-Level Generative Adversarial Networks for Semantic-Guided Scene Generation](https://mlsanity.com/paper/1912.12215/) (2019)
- [Mask-Guided Portrait Editing with Conditional GANs](https://mlsanity.com/paper/1905.10346/) (2019)
- [Drop the GAN: In Defense of Patches Nearest Neighbors as Single Image Generative Models](https://mlsanity.com/paper/2103.15545/) (2021)
- [Diverse Semantic Image Synthesis via Probability Distribution Modeling](https://mlsanity.com/paper/2103.06878/) (2021)
- [Are Perceptually-Aligned Gradients a General Property of Robust Classifiers?](https://mlsanity.com/paper/1910.08640/) (2019)
- [IMAGINE: Image Synthesis by Image-Guided Model Inversion](https://mlsanity.com/paper/2104.05895/) (2021)
- [Bridging Adversarial Robustness and Gradient Interpretability](https://mlsanity.com/paper/1903.11626/) (2019)
- [On the Benefits of Models with Perceptually-Aligned Gradients](https://mlsanity.com/paper/2005.01499/) (2020)

## Built on by (most cited first, 6 of 6)

- [Shedding More Light on Robust Classifiers under the lens of Energy-based Models](https://mlsanity.com/paper/2407.06315/) (2024)
- [Exploring the Connection between Robust and Generative Models](https://mlsanity.com/paper/2304.04033/) (2023)
- [pSTarC: Pseudo Source Guided Target Clustering for Fully Test-Time Adaptation](https://mlsanity.com/paper/2309.00846/) (2023)
- [Understanding Adversarial Training with Energy-based Models](https://mlsanity.com/paper/2505.22486/) (2025)
- [A Provable Energy-Guided Test-Time Defense Boosting Adversarial Robustness of Large Vision-Language Models](https://mlsanity.com/paper/2603.26984/) (2026)
- [Environment Maps Editing using Inverse Rendering and Adversarial Implicit Functions](https://mlsanity.com/paper/2410.18622/) (2024)

## Abstract (arXiv)

We offer a method for one-shot mask-guided image synthesis that allows controlling manipulations of a single image by inverting a quasi-robust classifier equipped with strong regularizers. Our proposed method, entitled MAGIC, leverages structured gradients from a pre-trained quasi-robust classifier to better preserve the input semantics while preserving its classification accuracy, thereby guaranteeing credibility in the synthesis. Unlike current methods that use complex primitives to supervise the process or use attention maps as a weak supervisory signal, MAGIC aggregates gradients over the input, driven by a guide binary mask that enforces a strong, spatial prior. MAGIC implements a series of manipulations with a single framework achieving shape and location control, intense non-rigid shape deformations, and copy/move operations in the presence of repeating objects and gives users firm control over the synthesis by requiring to simply specify binary guide masks. Our study and findings are supported by various qualitative comparisons with the state-of-the-art on the same images sampled from ImageNet and quantitative analysis using machine perception along with a user survey of 100+ participants that endorse our synthesis quality. Project page at https://mozhdehrouhsedaghat.github.io/magic.html. Code is available at https://github.com/mozhdehrouhsedaghat/magic