# Multimodality-guided Image Style Transfer using Cross-modal GAN Inversion

arXiv: https://arxiv.org/abs/2312.01671 · HTML: https://mlsanity.com/paper/2312.01671/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 51 references that are in the map.
- First posted on arXiv 4 Dec 2023.

## Builds on (most cited first, 30 of 46)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?](https://mlsanity.com/paper/1904.03189/) (2019)
- [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017)
- [DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation](https://mlsanity.com/paper/2110.02711/) (2021)
- [A Neural Algorithm of Artistic Style](https://mlsanity.com/paper/1508.06576/) (2015)
- [Generative Visual Manipulation on the Natural Image Manifold](https://mlsanity.com/paper/1609.03552/) (2016)
- [A Learned Representation For Artistic Style](https://mlsanity.com/paper/1610.07629/) (2016)
- [VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance](https://mlsanity.com/paper/2204.08583/) (2022)
- [Improved Texture Networks: Maximizing Quality and Diversity in Feed-forward Stylization and Texture Synthesis](https://mlsanity.com/paper/1701.02096/) (2017)
- [HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing](https://mlsanity.com/paper/2111.15666/) (2021)
- [CLIPstyler: Image Style Transfer with a Single Text Condition](https://mlsanity.com/paper/2112.00374/) (2021)
- [Arbitrary Style Transfer with Style-Attentional Networks](https://mlsanity.com/paper/1812.02342/) (2018)
- [AdaAttN: Revisit Attention Mechanism in Arbitrary Neural Style Transfer](https://mlsanity.com/paper/2108.03647/) (2021)

## Built on by (most cited first, 4 of 4)

- [Style-Specific Neurons for Steering LLMs in Text Style Transfer](https://mlsanity.com/paper/2410.00593/) (2024)
- [AIComposer: Any Style and Content Image Composition via Feature Integration](https://mlsanity.com/paper/2507.20721/) (2025)
- [ObjMST: An Object-Focused Multimodal Style Transfer Framework](https://mlsanity.com/paper/2503.04353/) (2025)
- [COT-AD: Cotton Analysis Dataset](https://mlsanity.com/paper/2507.18532/) (2025)

## Abstract (arXiv)

Image Style Transfer (IST) is an interdisciplinary topic of computer vision and art that continuously attracts researchers' interests. Different from traditional Image-guided Image Style Transfer (IIST) methods that require a style reference image as input to define the desired style, recent works start to tackle the problem in a text-guided manner, i.e., Text-guided Image Style Transfer (TIST). Compared to IIST, such approaches provide more flexibility with text-specified styles, which are useful in scenarios where the style is hard to define with reference images. Unfortunately, many TIST approaches produce undesirable artifacts in the transferred images. To address this issue, we present a novel method to achieve much improved style transfer based on text guidance. Meanwhile, to offer more flexibility than IIST and TIST, our method allows style inputs from multiple sources and modalities, enabling MultiModality-guided Image Style Transfer (MMIST). Specifically, we realize MMIST with a novel cross-modal GAN inversion method, which generates style representations consistent with specified styles. Such style representations facilitate style transfer and in principle generalize any IIST methods to MMIST. Large-scale experiments and user studies demonstrate that our method achieves state-of-the-art performance on TIST task. Furthermore, comprehensive qualitative results confirm the effectiveness of our method on MMIST task and cross-modal style interpolation.