mlsanityData to 7 Oct 2026

SynArtifact: Classifying and Alleviating Artifacts in Synthetic Images via Vision-Language Model

arXiv 2402.18068 ↗First posted 28 Feb 2024cs.CVcited by 15

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

In the rapidly evolving area of image synthesis, a serious challenge is the presence of complex artifacts that compromise perceptual realism of synthetic images. To alleviate artifacts and improve quality of synthetic images, we fine-tune Vision-Language Model (VLM) as artifact classifier to automatically identify and classify a wide range of artifacts and provide supervision for further optimizing generative models. Specifically, we develop a comprehensive artifact taxonomy and construct a dataset of synthetic images with artifact annotations for fine-tuning VLM, named SynArtifact-1K. The fine-tuned VLM exhibits superior ability of identifying artifacts and outperforms the baseline by 25.66%. To our knowledge, this is the first time such end-to-end artifact classification task and solution have been proposed. Finally, we leverage the output of VLM as feedback to refine the generative model for alleviating artifacts. Visualization results and user study demonstrate that the quality of images synthesized by the refined diffusion model has been obviously improved.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 40 references that are in the map. First posted on arXiv 28 Feb 2024.

Builds on

34 of 40 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 33

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  11. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  12. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
21 more
  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  2. 2016Improved Techniques for Training GANscited by 2,938
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  4. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  5. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  6. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  7. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
  8. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  9. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  10. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  11. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
  12. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  13. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 431
  14. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  15. 2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Modelscited by 376
  16. 2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generationcited by 253
  17. 2023AGIQA-3K: An Open Database for AI-Generated Image Quality Assessmentcited by 142
  18. 2023Depicting Beyond Scores: Advancing Image Quality Assessment through Multi-modal Language Modelscited by 104
  19. 2023A Perceptual Quality Assessment Exploration for AIGC Imagescited by 54
  20. 2023Visual Programming for Text-to-Image Generation and Evaluationcited by 53
  21. 2023X-IQE: eXplainable Image Quality Evaluation for Text-to-Image Generation with Visual Large Language Modelscited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 4; 2026: 5.

15 papers.

Sort:

By area:

  1. 2024Good Seed Makes a Good Crop: Discovering Secret Seeds in Text-to-Image Diffusion Modelscited by 47
  2. 2025Ivy-Fake: A Unified Explainable Framework and Benchmark for Image and Video AIGC Detectioncited by 34
  3. 2025Whitened CLIP as a Likelihood Surrogate of Images and Captionscited by 16
  4. 2024Detecting Human Artifacts from Text-to-Image Modelscited by 12
  5. 2025DiffDoctor: Diagnosing Image Diffusion Models Before Treatingcited by 7
  6. 2026Agentic Retoucher for Text-To-Image Generationcited by 7
  7. 2024Refine-by-Align: Reference-Guided Artifacts Refinement through Semantic Alignmentcited by 5
  8. 2024Synthetic Photography Detection: A Visual Guidance for Identifying Synthetic Images Created by AIcited by 3
  9. 2026Unveiling Perceptual Artifacts: A Fine-Grained Benchmark for Interpretable AI-Generated Image Detectioncited by 3
  10. 2024COBRA: COmBinatorial Retrieval Augmentation for Few-Shot Adaptationcited by 2
  11. 2025GeneVA: A Dataset of Human Annotations for Generative Text to Video Artifactscited by 2
  12. 2026Veritas++: Value-aware On-Policy Distillation for Perception-Enhanced AIGI Detectioncited by 2
3 more
  1. 2026GenShield: Unified Detection and Artifact Correction for AI-Generated Imagescited by 1
  2. 2024Synthesis and Perceptual Scaling of High Resolution Naturalistic Images Using Stable Diffusioncited by 0
  3. 2026VLM- and LLM-Driven Multi-Agent System for PET Image Denoisingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SalArt-VQA: Diagnosing Whether VLMs Understand Salient Artifacts in Generated Images
  2. 2025Synthesizing Artifact Dataset for Pixel-level Detection
  3. 2026See and Fix the Flaws: Enabling VLMs and Diffusion Models to Comprehend Visual Artifacts via Agentic Data Synthesis
  4. 2026Artifact-Bench: Evaluating MLLMs on Detecting and Assessing the Artifacts of AI-Generated Videos
  5. 2025Mirage: Unveiling Hidden Artifacts in Synthetic Images with Large Vision-Language Models
  6. 2025LEGION: Learning to Ground and Explain for Synthetic Image Detection
  7. 2024Bi-LORA: A Vision-Language Approach for Synthetic Image Detection
  8. 2025Synthetic Stimuli, Real Gains: Rethinking VLM Fine-Tuning Through Fully Controlled Data Generation