# Fine-grained Cross-modal Fusion based Refinement for Text-to-Image Synthesis

arXiv: https://arxiv.org/abs/2302.08706 · HTML: https://mlsanity.com/paper/2302.08706/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 51 references that are in the map.
- First posted on arXiv 17 Feb 2023.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Generative Adversarial Text to Image Synthesis](https://mlsanity.com/paper/1605.05396/) (2016)
- [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017)
- [StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1612.03242/) (2016)
- [DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesis](https://mlsanity.com/paper/1904.01310/) (2019)
- [cGANs with Projection Discriminator](https://mlsanity.com/paper/1802.05637/) (2018)
- [StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1710.10916/) (2017)
- [Controllable Text-to-Image Generation](https://mlsanity.com/paper/1909.07083/) (2019)
- [MirrorGAN: Learning Text-to-image Generation by Redescription](https://mlsanity.com/paper/1903.05854/) (2019)
- [Modulating early visual processing by language](https://mlsanity.com/paper/1707.00683/) (2017)
- [DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesis](https://mlsanity.com/paper/2008.05865/) (2020)
- [Learning What and Where to Draw](https://mlsanity.com/paper/1610.02454/) (2016)
- [Photographic Text-to-Image Synthesis with a Hierarchically-nested Adversarial Network](https://mlsanity.com/paper/1802.09178/) (2018)
- [Tell, Draw, and Repeat: Generating and Modifying Images Based on Continual Linguistic Instruction](https://mlsanity.com/paper/1811.09845/) (2018)
- [Semantics Disentangling for Text-to-Image Generation](https://mlsanity.com/paper/1904.01480/) (2019)
- [Survey on Deep Multi-modal Data Analytics: Collaboration, Rivalry and Fusion](https://mlsanity.com/paper/2006.08159/) (2020)
- [DAE-GAN: Dynamic Aspect-aware GAN for Text-to-Image Synthesis](https://mlsanity.com/paper/2108.12141/) (2021)
- [Context-Aware Layout to Image Generation with Enhanced Object Appearance](https://mlsanity.com/paper/2103.11897/) (2021)
- [Switchable Online Knowledge Distillation](https://mlsanity.com/paper/2209.04996/) (2022)
- [Iterative Views Agreement: An Iterative Low-Rank based Structured Optimization Method to Multi-View Spectral Clustering](https://mlsanity.com/paper/1608.05560/) (2016)
- [DTGAN: Dual Attention Generative Adversarial Networks for Text-to-Image Generation](https://mlsanity.com/paper/2011.02709/) (2020)
- [Delving Globally into Texture and Structure for Image Inpainting](https://mlsanity.com/paper/2209.08217/) (2022)
- [Cycle-Consistent Deep Generative Hashing for Cross-Modal Retrieval](https://mlsanity.com/paper/1804.11013/) (2018)
- [TIME: Text and Image Mutual-Translation Adversarial Networks](https://mlsanity.com/paper/2005.13192/) (2020)
- [Realistic Image Generation using Region-phrase Attention](https://mlsanity.com/paper/1902.05395/) (2019)

## Built on by (most cited first, 6 of 6)

- [DyFo: A Training-Free Dynamic Focus Visual Search for Enhancing LMMs in Fine-Grained Visual Understanding](https://mlsanity.com/paper/2504.14920/) (2025)
- [PosterO: Structuring Layout Trees to Enable Language Models in Generalized Content-Aware Layout Generation](https://mlsanity.com/paper/2505.07843/) (2025)
- [ConMo: Controllable Motion Disentanglement and Recomposition for Zero-Shot Motion Transfer](https://mlsanity.com/paper/2504.02451/) (2025)
- [ResVG: Enhancing Relation and Semantic Understanding in Multiple Instances for Visual Grounding](https://mlsanity.com/paper/2408.16314/) (2024)
- [Exploring and Exploiting the Asymmetric Valley of Deep Neural Networks](https://mlsanity.com/paper/2405.12489/) (2024)
- [Learning Attribute-Aware Hash Codes for Fine-Grained Image Retrieval via Query Optimization](https://mlsanity.com/paper/2509.17049/) (2025)

## Abstract (arXiv)

Text-to-image synthesis refers to generating visual-realistic and semantically consistent images from given textual descriptions. Previous approaches generate an initial low-resolution image and then refine it to be high-resolution. Despite the remarkable progress, these methods are limited in fully utilizing the given texts and could generate text-mismatched images, especially when the text description is complex. We propose a novel Fine-grained text-image Fusion based Generative Adversarial Networks, dubbed FF-GAN, which consists of two modules: Fine-grained text-image Fusion Block (FF-Block) and Global Semantic Refinement (GSR). The proposed FF-Block integrates an attention block and several convolution layers to effectively fuse the fine-grained word-context features into the corresponding visual features, in which the text information is fully used to refine the initial image with more details. And the GSR is proposed to improve the global semantic consistency between linguistic and visual features during the refinement process. Extensive experiments on CUB-200 and COCO datasets demonstrate the superiority of FF-GAN over other state-of-the-art approaches in generating images with semantic consistency to the given texts.Code is available at https://github.com/haoranhfut/FF-GAN.