mlsanityData to 7 Oct 2026

N"UWA-LIP: Language Guided Image Inpainting with Defect-free VQGAN

arXiv 2202.05009 ↗First posted 10 Feb 2022cs.CVcited by 19

Not on the map. Why?

Abstract (arXiv)

Language guided image inpainting aims to fill in the defective regions of an image under the guidance of text while keeping non-defective regions unchanged. However, the encoding process of existing models suffers from either receptive spreading of defective regions or information loss of non-defective regions, giving rise to visually unappealing inpainting results. To address the above issues, this paper proposes N"UWA-LIP by incorporating defect-free VQGAN (DF-VQGAN) with multi-perspective sequence to sequence (MP-S2S). In particular, DF-VQGAN introduces relative estimation to control receptive spreading and adopts symmetrical connections to protect information. MP-S2S further enhances visual information from complementary perspectives, including both low-level pixels and high-level tokens. Experiments show that DF-VQGAN performs more robustness than VQGAN. To evaluate the inpainting performance of our model, we built up 3 open-domain benchmarks, where N"UWA-LIP is also superior to recent strong baselines.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 29 of its 44 references that are in the map. First posted on arXiv 10 Feb 2022.

Builds on

29 of 44 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  8. 2017Neural Discrete Representation Learningcited by 4,650
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  11. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  12. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
17 more
  1. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  2. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  3. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  4. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  5. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  6. 2021Resolution-robust Large Mask Inpainting with Fourier Convolutionscited by 730
  7. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  8. 2018Free-Form Image Inpainting with Gated Convolutioncited by 594
  9. 2018Image Inpainting for Irregular Holes Using Partial Convolutionscited by 505
  10. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  11. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  12. 2021Large Scale Image Completion via Co-Modulated Generative Adversarial Networkscited by 219
  13. 2019Pluralistic Image Completioncited by 162
  14. 2021ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesiscited by 122
  15. 2021Paint by Wordcited by 107
  16. 2021Generating Diverse Structure for Image Inpainting With Hierarchical VQ-VAEcited by 106
  17. 2021The Image Local Autoregressive Transformercited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 8; 2025: 3; 2026: 3.

19 papers.

Sort:

By area:

  1. 2023Controlling Vision-Language Models for Multi-Task Image Restorationcited by 108
  2. 2024Scaling the Codebook Size of VQGAN to 100,000 with a Utilization Rate of 99%cited by 68
  3. 2023HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image Inpainting with Diffusion Modelscited by 67
  4. 2024Deep Learning-based Image and Video Inpainting: A Surveycited by 44
  5. 2023TextIR: A Simple Framework for Text-based Editable Image Restorationcited by 17
  6. 2023Vision + Language Applications: A Surveycited by 8
  7. 2025CODA: Repurposing Continuous VAEs for Discrete Tokenizationcited by 8
  8. 2024Sketch-guided Image Inpainting with Partial Discrete Diffusion Processcited by 7
  9. 2025Discrete Tokenization for Multimodal LLMs: A Comprehensive Surveycited by 7
  10. 2023Architext: Language-Driven Generative Architecture Designcited by 5
  11. 2024Responsible Visual Editingcited by 2
  12. 2024GroupDiff: Diffusion-based Group Portrait Editingcited by 2
7 more
  1. 2024DAFT-GAN: Dual Affine Transformation Generative Adversarial Network for Text-Guided Image Inpaintingcited by 1
  2. 2026Distributional Matching for Vector Quantization: A Unified Theoretical and Empirical Frameworkcited by 1
  3. 2026VQ-Transplant: Efficient VQ-Module Integration for Pre-trained Visual Tokenizerscited by 1
  4. 2024Context-aware Talking Face Video Generationcited by 0
  5. 2024FaithFill: Faithful Inpainting for Object Completion Using a Single Reference Imagecited by 0
  6. 2025Recovering Partially Corrupted Objects via Sketch-Guided Bidirectional Feature Interactioncited by 0
  7. 2026ArcVQ-VAE: A Spherical Vector Quantization Framework with ArcCosine Additive Margincited by 0