mlsanityData to 7 Oct 2026

How Do Inpainting Artifacts Propagate to Language?

arXiv 2602.20520 ↗First posted 24 Feb 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Image editing and quality assessment

Abstract (arXiv)

We study how visual artifacts introduced by diffusion-based inpainting affect language generation in vision-language models. We use a two-stage diagnostic setup in which masked image regions are reconstructed and then provided to captioning models, enabling controlled comparisons between captions generated from original and reconstructed inputs. Across multiple datasets, we analyze the relationship between reconstruction fidelity and downstream caption quality. We observe consistent associations between pixel-level and perceptual reconstruction metrics and both lexical and semantic captioning performance. Additional analysis of intermediate visual representations and attention patterns shows that inpainting artifacts lead to systematic, layer-dependent changes in model behavior. Together, these results provide a practical diagnostic framework for examining how visual reconstruction quality influences language generation in multimodal systems.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Image editing and quality assessment, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 15 of its 30 references that are in the map. First posted on arXiv 24 Feb 2026.

Builds on

15 of 30 references.

References · 15

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  5. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  6. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  7. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  8. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  9. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  10. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  11. 2016Modeling Context in Referring Expressionscited by 1,054
  12. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
3 more
  1. 2018Object Hallucination in Image Captioningcited by 661
  2. 2023LP-MusicCaps: LLM-Based Pseudo Music Captioningcited by 83
  3. 2024ROCOv2: Radiology Objects in COntext Version 2, an Updated Multimodal Image Datasetcited by 54

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026UniCanvas: A Diffusion-base Unified Model for Text-in-Image Joint Generation
  2. 2024Image Regeneration: Evaluating Text-to-Image Model via Generating Identical Image with Multimodal Large Language Models
  3. 2025Text Speaks Louder than Vision: ASCII Art Reveals Textual Biases in Vision-Language Models
  4. 2023Localizing and Editing Knowledge in Text-to-Image Generative Models
  5. 2026Language-Free Generative Editing from One Visual Example
  6. 2024Is Your Text-to-Image Model Robust to Caption Noise?
  7. 2024OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation
  8. 2025The Describe-Then-Generate Bottleneck: How VLM Descriptions Alter Image Generation Outcomes