# Improving Cross-modal Alignment for Text-Guided Image Inpainting

arXiv: https://arxiv.org/abs/2301.11362 · HTML: https://mlsanity.com/paper/2301.11362/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 41 references that are in the map.
- First posted on arXiv 26 Jan 2023.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [SimVLM: Simple Visual Language Model Pretraining with Weak Supervision](https://mlsanity.com/paper/2108.10904/) (2021)
- [PointRend: Image Segmentation as Rendering](https://mlsanity.com/paper/1912.08193/) (2019)
- [In-Domain GAN Inversion for Real Image Editing](https://mlsanity.com/paper/2004.00049/) (2020)
- [Pluralistic Image Completion](https://mlsanity.com/paper/1903.04227/) (2019)
- [High-Fidelity Pluralistic Image Completion with Transformers](https://mlsanity.com/paper/2103.14031/) (2021)
- [StructureFlow: Image Inpainting via Structure-aware Appearance Flow](https://mlsanity.com/paper/1908.03852/) (2019)
- [Coherent Semantic Attention for Image Inpainting](https://mlsanity.com/paper/1905.12384/) (2019)
- [Image Inpainting via Conditional Texture and Structure Dual Generation](https://mlsanity.com/paper/2108.09760/) (2021)
- [ClarET: Pre-training a Correlation-Aware Context-To-Event Transformer for Event-Centric Generation and Classification](https://mlsanity.com/paper/2203.02225/) (2022)
- [A Formal Evaluation of PSNR as Quality Measurement Parameter for Image Segmentation Algorithms](https://mlsanity.com/paper/1605.07116/) (2016)
- [EventBERT: A Pre-Trained Model for Event Correlation Reasoning](https://mlsanity.com/paper/2110.06533/) (2021)

## Built on by (most cited first, 21 of 21)

- [Exploiting Diffusion Prior for Out-of-Distribution Detection](https://mlsanity.com/paper/2406.11105/) (2024)
- [Bridging the Gap between Synthetic and Authentic Images for Multimodal Machine Translation](https://mlsanity.com/paper/2310.13361/) (2023)
- [Educational Personalized Learning Path Planning with Large Language Models](https://mlsanity.com/paper/2407.11773/) (2024)
- [Recent Advances in Transformer and Large Language Models for UAV Applications](https://mlsanity.com/paper/2508.11834/) (2025)
- [High-Fidelity Pseudo-label Generation by Large Language Models for Training Robust Radiology Report Classifiers](https://mlsanity.com/paper/2505.01693/) (2025)
- [Enhancing Agricultural Machinery Management through Advanced LLM Integration](https://mlsanity.com/paper/2407.20588/) (2024)
- [DAFT-GAN: Dual Affine Transformation Generative Adversarial Network for Text-Guided Image Inpainting](https://mlsanity.com/paper/2408.04962/) (2024)
- [Large Language Models for Zero-Shot Multicultural Name Recognition](https://mlsanity.com/paper/2507.04149/) (2025)
- [DeCoT: Decomposing Complex Instructions for Enhanced Text-to-Image Generation with Large Language Models](https://mlsanity.com/paper/2508.12396/) (2025)
- [M3PO: Multimodal-Model-Guided Preference Optimization for Visual Instruction Following](https://mlsanity.com/paper/2508.12458/) (2025)
- [LLMs for Enhanced Agricultural Meteorological Recommendations](https://mlsanity.com/paper/2408.04640/) (2024)
- [Enhancing AI-Driven Psychological Consultation: Layered Prompts with Large Language Models](https://mlsanity.com/paper/2408.16276/) (2024)
- [Exploring Large Vision-Language Models for Robust and Efficient Industrial Anomaly Detection](https://mlsanity.com/paper/2412.00890/) (2024)
- [Bridging Vision and Language: Modeling Causality and Temporality in Video Narratives](https://mlsanity.com/paper/2412.10720/) (2024)
- [Hierarchical Vision-Language Alignment for Text-to-Image Generation via Diffusion Models](https://mlsanity.com/paper/2501.00917/) (2025)
- [Vision-Driven Prompt Optimization for Large Language Models in Multimodal Generative Tasks](https://mlsanity.com/paper/2501.02527/) (2025)
- [Cross-Domain Semantic Segmentation with Large Language Model-Assisted Descriptor Generation](https://mlsanity.com/paper/2501.16467/) (2025)
- [A Generative Framework for Bidirectional Image-Report Understanding in Chest Radiography](https://mlsanity.com/paper/2502.05926/) (2025)
- [LVLM-Composer's Explicit Planning for Image Generation](https://mlsanity.com/paper/2507.04152/) (2025)
- [VisuCraft: Enhancing Large Vision-Language Models for Complex Visual-Guided Creative Content Generation via Structured Information Extraction](https://mlsanity.com/paper/2508.02890/) (2025)
- [Spectral Origins of the Self-Correction Blind Spot in Autoregressive Generation](https://mlsanity.com/paper/2607.09803/) (2026)

## Abstract (arXiv)

Text-guided image inpainting (TGII) aims to restore missing regions based on a given text in a damaged image. Existing methods are based on a strong vision encoder and a cross-modal fusion model to integrate cross-modal features. However, these methods allocate most of the computation to visual encoding, while light computation on modeling modality interactions. Moreover, they take cross-modal fusion for depth features, which ignores a fine-grained alignment between text and image. Recently, vision-language pre-trained models (VLPM), encapsulating rich cross-modal alignment knowledge, have advanced in most multimodal tasks. In this work, we propose a novel model for TGII by improving cross-modal alignment (CMA). CMA model consists of a VLPM as a vision-language encoder, an image generator and global-local discriminators. To explore cross-modal alignment knowledge for image restoration, we introduce cross-modal alignment distillation and in-sample distribution distillation. In addition, we employ adversarial training to enhance the model to fill the missing region in complicated structures effectively. Experiments are conducted on two popular vision-language datasets. Results show that our model achieves state-of-the-art performance compared with other strong competitors.